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SUMMARY 


Objective* 

The  objectives  were  (a)  to  examine  alternative  approaches  for  Air  Combat  Maneuvering  Performance  Measurement 
(ACMPM)  based  on  a  state-space  measurement  concept  called  TACSPACE,  (b)  to  develop  a  functional  specification  by 
which  performance  measurement  information  from  free  engagements  could  be  displayed  on  the  Simulator  for  Air-to-Air 
Combat  (SA  AC)  console  for  the  benefit  of  the  Instructor  Pilot  (IP),  and  (c)  to  examine  the  feasibility  and  efficacy  of  utilizing 
such  a  system  on  the  Air  Force  Air  Combat  Maneuvering  Instrumentation  (ACMI)  or  the  Navy  Tactical  Air  Combat 
Training  System  (TACTS). 


Background/Rationale 

Measurement  of  pilot  performance  during  free  air  combat  engagements  provides  a  unique  and  important 
measurement  challenge.  Existing  automated  performance  measurement  structures  that  compare  the  aircraft’s 
maneuvering  with  a  standard  profile,  or  that  examine  only  mission  success,  are  of  limited  utility  in  this  application. 

It  was  hypothesized  that  a  measurement  structure  which  could  examine  interactions  between  relative  aircraft  position 
and  pilot  control  behavior  would  provide  improved  measurement  capability.  The  TACSPACE  model  provides  such  a 
structure. 


Approach 

A  two-stage  approach  was  taken:  (a)  specification  and  testing  of  candidate  measurement  models,  and  (b) 
development  of  a  functional  specification  for  candidate  display  concepts.  Multivariate  discriminant  analyses  of  the  data 
were  computed,  and  an  algorithm  was  developed.  The  algorithm  was  then  applied  to  the  TACSPACE  concept  to  test 
its  validity,  generalization,  and  efficacy  using  the  existing  data  base. 


Specifics 

Hypotheses  were  developed  about  candidate  TACSPACE  related  measurement  structures  and  tested  using  a  data 
base  of  time  history  records.  Several  different  ACMPM  models  varying  greatly  in  complexity  were  developed  and  tested. 
A  refined  version  of  TACSPACE  was  then  constructed.  Analyses  of  pilot  control  behavior  were  then  superimposed  over 
the  TACSPACE  structure.  Finally,  a  TACSPACE  structure,  including  only  three  regions  (offensive,  neutral,  and 
defensive)  with  overlaid  pilot  control  measures,  was  evaluated.  Analyses  were  also  computed  using  only  measures 
available  from  ACMI/TACTS.  Training  on  the  SAAC  was  observed,  and  SAAC  instructional,  administrative,  and 
engineering  personnel  were  interviewed. 

Large  differences  were  found  between  the  models  in  their  ability  to  account  for  performance  variance.  The 
TACSPACE  structure  was  reduced  from  616  individual  cells  (identified  by  aspect  angle,  line  of  sight,  and  range)  to  80 
cells.  This  combining  of  the  cells  facilitated  analyses  and  at  the  same  time  provided  a  more  realistic  working  model  for 
the  user. 


Conelusions/Recommendations 

The  TACSPACE  concept  provides  a  useful  framework  on  which  to  build  an  ACMPM  system.  However,  additional 
information  should  be  included  and  both  further  development  and  validation  are  required.  The  TACSPACE  and  data- 
analysis  procedures  that  were  developed  provide  a  reasonable  method  to  assess  overall  maneuvering  performance. 
However,  in  their  present  form,  the  structure  and  resulting  equations  provide  only  indirect  performance  diagnosis.  For 
direct  diagnosis  in  a  language  that  instructor  pilots  could  use,  future  work  should  consider  combining  TACSPACE  with 
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the  evolving  technology  of  expert  systems  by  inserting  some  decision  rules  to  guide  the  selection  of  subsequent 
measurement  within  appropriate  TACSPACE  cells. 

Although  real-time  implementation  on  the  SAAC  was  found  to  be  feasible,  its  usefulness  appeared  questionable 
due  to  the  lack  of  sufficient  diagnostic  information. 

The  TACSPACE  concept  was  found  to  be  applicable  to  the  ACMI/TACTS  with  very  little  loss  of  predictive  validity 
as  a  result  of  the  reduced  number  of  available  measures. 


TABLE  OF  CONTENTS 


Section  Page 


I  INTRODUCTION .  5 

II  DATA  COLLECTION .  10 

Analytical  Phase .  10 

Empirical  Data  Collection  Phase  .  12 

III  ANALYSIS  METHOD .  14 

Segment  Engagements  .  16 

Perform  TACSPACE  Calculations  .  19 

Nonparametric  Discriminant  Analysis  .  22 

Suitability  Decisions  .  22 

IV  ANALYSIS  RESULTS .  23 

Whole  Engagement  Analysis,  Similar  Aircraft  .  23 

Neutral  Segment  Analyses,  Similar  Aircraft .  32 

Offensive  and  Defensive  TACSPACE  Analysis,  Similar 

Aircraft .  33 

Summary  of  Similar  Aircraft  .  35 

Reduced  Measure  Set  Analysis  For  ACM  I /TACTS .  36 

Dissimilar  Aircraft  Analysis .  37 

Weapon  Envelope  Study  .  .  37 

V  DEVELOPMENT  OF  FUNCTIONAL  SPECIFICATIONS .  40 

Current  Training .  40 

Implementation  Constraints.  . .  41 

Design  Approach  Adopted  .  42 

VI  CONCLUSIONS .  45 

REFERENCES .  47 

APPENDIX  A.  DISCUSSION  OF  STATISTICAL  PROCEDURES  ....  48 

Previous  Work .  48 

T-Test .  48 

Tukey's  Quick  Test  of  Location .  49 

Unit-Scaling  Technique .  50 

APPENDIX  B.  SPECIFICATIONS .  51 

Scope .  51 

Flow  Diagram .  51 

Parameter  Sampling .  51 

TACSPACE  Computations . 51 

Computations  During  Each  Sampling  Cycle  .  55 

End-of-Engagement  Computations .  56 

Displays.  . .  56 

APPENDIX  C.  DATA  FOR  DISCRIMINANT  MODELS .  59 

APPENDIX  D.  NORMATIVE  DATA .  83 


1 


LIST  OF  TABLES 


Table  No. 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

I 

14 
B1 

B2 

I 

B3 

B4 

B5 

I 

Cl 


CANDIDATE  TIME -HISTORY  SEGMENT  SUMMARY  MEASURES . 

CANDIDATE  TACSPACE  CELL  SUMMARY  MEASURES  . 

SIMILAR  AIRCRAFT  ANALYSES . 

WHOLE  ENGAGEMENT  TACSPACE  DISCRIMINANT  VARIABLE . 

CONTRAST  TABLE  OF  LOS  AND  ASPECT  FOR  WHOLE  ENGAGEMENT.  . 

LOW-RESOLUTION  WHOLE  ENGAGEMENT  TACSPACE  CELL  SUMMARY 
MODELS  . 

NEUTRAL  SEGMENT  TACSPACE  CELL  SUMMARY  MEASURES  . 

OFFENSIVE  SEGMENT  TACSPACE  CELL  SUMMARY  MEASURES  .... 

DEFENSIVE  SEGMENT  TACSPACE  CELL  SUMMARY  MEASURES  .... 

SUMMARY  OF  SIMILAR  AIRCRAFT  ANALYSES  . 

DEFENSIVE  SEGMENT  REDUCED  TACSPACE  CELL  SUMMARY  MEASURES 
SET . 

OFFENSIVE  SEGMENT  REDUCED  TACSPACE  CELL  SUMMARY  MEASURES 

KILL  ENVELOPE  ANALYSIS  VARIABLES  . 

KILL  ENVELOPE  STUDY  RESULTS . 

BASIC  PARAMETERS  . 

NUMBERING  OF  TACSPACE  CELLS  . 

COMPUTATIONS  TO  BE  MADE  DURING  EACH  SAMPLING  CYCLE  .  .  . 

TRANSFORMATIONS . 

DISCRIMINANT  SCORE  COMPUTATIONS . 

MULTIVARIATE  DISCRIMINANT  SEGMENT  SUMMARY  MODEL  FOR  WHOLE 
ENGAGEMENT  . 


C2 


UNIT-SCALED  TACSPACE  DISCRIMINANT  MODEL  FOR  WHOLE 
ENGAGEMENT  . 


LIST  OF  TABLES 


(continued) 


Table  No.  Page 

C3  MULTIVARIATE  DISCRIMINANT  SEGMENT  SUMMARY  MODEL  FOR 

NEUTRAL  SEGMENT . 67 

C4  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL  FOR  NEUTRAL 

SEGMENT . 68 

C5  UNIT  SCALED  SEGMENT  SUMMARY  MODEL  FOR  OFFENSIVE  SEGMENT.  .  72 

C6  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL  FOR  OFFENSIVE 

SEGMENT . 73 

C7  UNIT  SCALED  SEGMENT  SUMMARY  MODEL  FOR  DEFENSIVE  SEGMENT.  .  74 

C8  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL  FOR  DEFENSIVE 

SEGMENT . 75 

C9  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  SUMMARY  SEGMENT  MODEL  FOR 

REDUCED  ACMI /TACTS  WHOLE  ENGAGMENT  .  76 

CIO  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE  DISCRIMINANT 

MODEL  FOR  WHOLE  ENGAGEMENT . 77 

Dl  NEUTRAL  SEGMENT  SUMMARY  MEASURES:  MEANS  AND  STANDARD 

DEVIATION . 84 

D2  OFFENSIVE  SEGMENT  SUMMARY  MEASURES:  MEANS  AND  STANDARD 

DEVIATIONS . 85 

D3  DEFENSIVE  SEGMENT  SUMMARY  MEASURES:  MEANS  AND  STANDARD 

DEVIATION . 86 

D4  OFFENSIVE  CELL  OCCUPANCY:  MEANS  AND  STANDARD  DEVIATIONS  .  87 

D5  DEFENSIVE  CELL  OCCUPANCY:  MEANS  AND  STANDARD  DEVIATIONS  .  88 


3 


LIST  OF  FIGURES 


Figure  No.  Page 

1  ACM  Mission  Sequence .  7 

2  The  Air  Combat  Model:  Best  and  Worst  Cases .  8 

3  Relationships  Between  LOS,  ASPECT,  and  RANGE .  15 

4  TACSPACE .  17 

5  Flow  Diagram.  Analysis  Methods  .  18 

6  High  Resolution  TACSPACE .  25 

7  Low  Resolution  TACSPACE  .  28 

8  Final  TACSPACE  Model .  31 

9  Real-Time  Display  Format . 43 

10  Summary  Display  Format .  44 

B1  Top-Level  Flow  Diagram .  52 


4 


I.  INTRODUCTION 


Measurement  of  pilot  performance  during  free  air  combat  engagements 
provides  a  technical  challenge  that  is  at  once  unique  and  important.  Free 
air  combat  maneuvering  (ACM)  is  unique  in  terms  of  performance  measurement 
requirements  for  two  main  reasons.  First,  most  previous  aircrew  perfor¬ 
mance  measurement  work  has  been  conducted  on  relatively  simple  visual  or 
instrument  flight  maneuvers  for  which  a  desired  profile  has  been  defined 
and  is  known  by  both  the  pilot  and  the  performance  analyst.  It  is  a  com¬ 
paratively  simple  procedure  to  measure  error  from  the  established  profile 
and  to  use  various  transforms  of  this  error  as  measures  of  performance. 

In  free  ACM,  there  are  no  such  established  profiles  against  which  to 
measure;  the  reference  datum  is  continually  changing  and  performance  cri¬ 
teria  are  vague.  Some  existing  models,  such  as  the  Automated  Maneuvering 
Logic  (AML),  offer  a  logical  decision  network  that  attempts  to  simulate 
the  movements  of  an  aircraft  during  free  engagements.  No  claims  are  made 
that  these  either  provide  an  optimum  solution  representative  of  what  a 
skilled  pilot  does.  Thus,  any  attempt  to  measure  ACM  performance  against 
any  kind  of  derived  profile  is  probably  futile,  unless  a  representative 
sample  of  all  possible  profiles  is  considered  by  the  measurement  logic. 

It  may  be  feasible  to  develop  an  AML  type  system  based  on  a  knowledge 
representation  model  of  a  skilled  pilot.  This,  however,  has  never  been 
done.  Even  this  approach  may  not  capture  all  of  what  may  be  important 
elements  of  the  task  such  as  gamesmanship,  Intimidation,  and  faking.  It 
might,  however,  provide  a  good  foundation  for  a  future  diagnostic  air  com¬ 
bat  measurement  system  when  combined  with  the  approach  taken  in  this 
study. 

The  second  factor  that  makes  ACM  performance  measurement  different 
from  usual  aircrew  performance  measurement  techniques  is  that  most  perfor¬ 
mance  measurement  applications  involve  a  single  pilot  or  aircrew  pursuing 
a  recognizable  objective.  ACM  involves,  at  a  minimum,  a  pair  of  pilots 
and  aircraft  with  mutually  exclusive  objectives.  The  dynamic  relationship 
between  the  two  aircraft  is  constantly  changing  as  each  pilot  maneuvers  to 
counteract  the  maneuvering  of  the  other.  Any  valid  performance  measure¬ 
ment  system  must,  therefore,  take  into  account  the  maneuvering  of  both 
proponent  and  opponent  aircraft  in  order  to  obtain  an  accurate  index  of 
pilot  performance. 

Development  of  automated  capability  for  ACM  Performance  Measurement 
(ACMPM)  is  important  for  one  primary  reason.  The  dynamic  and  fast  moving 
actions  of  a  pair  or  a  number  of  pilots  and  aircraft  are  difficult  for  an 
instructor  pilot  (IP),  encumbered  by  normal  human  limitations,  to  perceive 
and  evaluate.  An  automated  ACM  performance  measurement  system  would  pro¬ 
vide  a  valuable  complement  to  the  IP's  own  abilities. 
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An  automated  system  for  ACMPM  requires  a  considerable  amount  of  data 
measurement,  transmission,  storage,  processing,  and  display  capability. 
Until  recently,  such  required  capability  could  only  be  found  in  ground 
based  flight  simulators  such  as  the  Simulator  for  Air-to-Air  Combat  (SAAC) 
at  Luke  AFB.  During  the  past  few  years,  however,  capabilities  for  the 
airborne  measurement  of  pilot  performance  have  been  developed  and  refined. 
The  Air  Force  Air  Combat  Maneuvering  Instrumentation  (ACMI)  and  Navy 
Tactical  Air  Combat  Training  System  (TACTS)  now  provide  the  basic  capabil¬ 
ities  required  for  an  ACMPM  system. 

A  number  of  recent  attempts  to  provide  an  ACMPM  system  have  been 
made.  DeBerg  (1977)  performed  a  factor  analytic  study  of  maneuvering  data 
obtained  during  simulated  ACM.  He  found  that  the  most  important  factor  in 
describing  ACM  involved  measures  of  energy  management. 

Simpson  and  Oberle  (1977)  developed  a  method  for  assessing  pilot  per¬ 
formance  during  ACM  based  largely  on  tracking  the  relative  positions  of 
the  two  or  more  aircraft  involved  in  a  fight.  This  ACMPM  method  uses 
measures  available  on  flight  simulators  as  well  as  on  ACMI/TACTS. 

The  Vought  Corporation  "Good  Stick  Index,"  developed  and  refined  by 
Moore,  Madison,  Sepp,  Stracener,  and  Coward  (1979),  was  developed  to 
assess  the  effects  of  air  combat  training.  The  GSI  measures  several 
aspects  of  the  performance  of  pilots  attacking  a  target  that  is  flying  a 
pre-defined  profile. 

Brlctson  and  his  associates  at  Dunlap  and  Associates,  Inc.,  Western 
Division,  have  created  a  model  of  ACM  engagements  based  on  a  hypothetical 
ideal  mission  sequence.  The  ACM  mission  sequence  (Figure  1)  includes  the 
chain  of  events  that  theoretically  occur  during  a  successful  engagement. 
Any  deviation  from  this  sequence  is  indicative  of  a  less  than  entirely 
successful  engagement.  Figure  2  shows  the  best  and  worst  possible  cases 
for  a  given  engagement.  Based  on  these  models,  the  Dunlap  and  Associates 
group  has  developed  a  performance  feedback  system  called  PACE  (e.g., 
Ciavarelll  (1980)).  Performance  assessment  is  provided  on  five  kinds  of 
jMasures: 

1.  Whether  radar  contact  is  obtained  and  the  range  between  air¬ 
craft. 

2.  Whether  visual  contact  is  obtained  and  the  range  between  air¬ 
craft. 

3.  Whether  the  pilot  took  the  first  shot  during  the  engagement. 

4.  The  aircraft  position  in  terms  of  weapons  envelope  when  weapon 
was  fired. 

5.  Win/ loss  as  a  function  of  weapon  type. 
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Radar  Procedures - —-Look-Out  Procedures  - —Tactics  *  Maneuvers  - —Envelope  Recognition 


Figure  1.  ACM  mission  sequence 
(from  Ciavarelli,  1980) 


Time  to  Time  to  Time  to  Time  to 

Contact  Tally  Ho  1st  Shot  Kill 


GOAL:  [GAIN  EARLY  ADVANTAGE,  FIRST  SHOT,  AND  QUICK  KILL  I 


Figure  2.  The  air  combat  model:  best  and  worst  cases, 
(from  Ciavarelli ,  1980) 
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The  PACE  system  collects  and  records  these  measures  at  both  Individual 
pilot  and  operational  unit  levels  of  summation.  This  allows  the  squadron 
commander  to  obtain  quickly  and  easily  a  global  picture  of  the  performance 
of  the  unit  as  compared  to  a  desired  standard  of  performance,  suggesting 
areas  for  future  training  concentration. 

In  contrast  to  the  rather  global  measures  of  performance  provided  by 
the  PACE  model,  Kelly,  Wooldridge,  Hennessy,  Vreuls,  Barnebey,  Cotton,  and 
Reed  (1979)  examined  in  considerable  detail  the  performance  of  the  pilot 
in  a  smaller  segment  of  the  mission  sequence,  that  associated  with  maneu¬ 
vering  to  the  lethal  zone  and  firing  a  missile.  Their  results  demonstrated 
the  feasibility  of  measuring  ACM  performance  using  detailed  performance 
measures  in  a  multivariate  measurement  structure. 

The  Kelly  et  al.  (1979)  report  suggested  several  improvements  that 
could  be  made  by  more  extensive  multivariate  data  analysis  techniques. 
This  current  technical  report  summarizes  the  results  of  data  analyses  per¬ 
formed  on  that  data  base  using  a  more  complex  data  analysis  structure.  It 
further  provides  a  functional  specification  which  could  be  used  to  imple¬ 
ment  a  performance  measurement  package  on  the  SAAC. 


II.  MTA  COLLECTION 


This  section  summarizes  the  procedures  used  to  collect  the  data  that 
were  analyzed  for  this  report.  This  is  the  same  database  used  for  analy¬ 
sis  in  the  earlier  Kelly  et  al.  (1979)  report,  which  provides  a  more 
detailed  description  of  data  collection  techniques. 

The  technical  approach  Involved  a  combined  analytical  and  empirical 
examination  of  the  performance  measurement  problem.  First,  an  examination 
of  the  ACM  task,  training  and  evaluation  techniques  provided  the  framework 
for  a  candidate  measurement  structure  and  a  list  of  possible  measures. 
Data  were  then  collected  during  ACM  free  engagements  and  the  measurement 
structures  were  applied  to  the  data. 

ANALYTICAL  PHASE 

During  the  analytical  phase  of  the  prior  effort,  information  was 
obtained  from  ACM  training  materials  and  from  interviews  with  academic, 
ISD,  and  Fighter  Weapons  School  personnel  as  well  as  from  line  ACM  instruc¬ 
tors  at  Luke  AFB.  From  these  information  sources,  hypotheses  were  developed 
about  global  differences  between  skilled  and  less  skilled  ACM  pilots. 
Then,  within  these  hypotheses,  lists  of  specific  measures  were  developed 
that  would  correlate  with  the  differences. 

The  analysis  revealed  several  different,  although  probably  not  ortho¬ 
gonal,  dimensions  of  ACM  performance  to  be  tested  for  inclusion  in  an 
A  CMPM  structure.  Also  listed  were  groups  of  measures  that  should  be 
correlated  with  each  of  the  global  dimensions.  These  were  summarized  by 
Kelly,  Wooldridge,  Hennessy,  and  Reed  (1979). 

Engagement  outcomes .  The  most  obvious  measure  of  ACM  skill  is  the 
number  of  kills  achieved  by  a  pilot  in  real  or  simulated  ACM  engagements. 
This  measure  is  probably  the  most  common  criterion  in  use  today.  Engage¬ 
ment  outcomes  can  be  contaminated,  however,  by  many  other  variables  (e.g., 
the  less  skilled  pilot  can  get  a  lucky  shot).  Other  variables  must, 
therefore,  be  examined  in  order  to  provide  a  fully  valid  measure  of  pilot 
performance. 

Energy  management.  It  is  axiomatic  that  pilots  who  manage  their  ki¬ 
netic  and  potential  energy  well  have  a  significant  advantage  over  pilots 
who  manage  their  energy  less  well.  A  factor  analytic  study  by  DeBerg 
(1977)  demonstrated  that  energy  measures  provide  the  most  important  class 
of  measures  to  be  used  in  describing  an  air  combat  engagement.  More 
recently,  Moroney,  Pruitt,  and  Lau  (1979)  recommended  the  use  of  energy 
maneuverability  displays  during  training  of  ACM. 

Although  this  class  of  measures  offers  attractive  results,  exsistlng 
models  that  combine  variables  of  potential  and  kinectic  energy  require 


sophisticated  computations.  Extensive  developmental  work  needs  to  be  done 
to  optimize  just  these  measures.  Inclusion  of  these  measures  would  serious¬ 
ly  have  effected  the  timeliness  of  this  study  and  limited  the  probability 
of  their  implementation  in  an  operational  simulator  or  instrumented  range 
system. 

Aggressiveness .  The  trait  most  often  cited  as  an  indicator  of  ACM 
performance  was  aggressiveness.  The  aggressive  pilot  spends  little  time 
in  a  neutral  position.  He  is  continually  maneuvering ,  making  the  opponent 
react  to  his  moves.  He  is  eager  to  get  into  the  fight  quickly  and  employs 
lead  pursuit  and  lead  turns  more  skillfully  when  moving  into  firing  posi¬ 
tion.  Some  experts  postulated  a  correlation  between  pilot  aggressiveness 
and  greater  use  of  the  roll  axis  and  of  the  vertical  dimension.  Measures 
of  turn  rate,  roll  rate,  plane  of  action  and  lateral  velocity  were  in¬ 
cluded  to  indicate  relative  aggresiveness. 

Situation  awareness.  Skilled  pilots  maintain  a  keen  awareness  of  the 
changing  situation  around  them.  They  know  where  they  are  in  relation  to 
the  terrain,  friendly  forces,  and  opposing  forces.  They  are  able  to  extra¬ 
polate  their  position  and  that  of  the  opponent  into  the  future.  The  pilots 
exhibiting  good  situation  awareness  are  probably  more  active  in  use  of 
controls  such  as  throttle  and  speedbrake  as  they  anticipate  the  opponent's 
moves.  They  are  aware  of  weapons  envelopes  and  anticipate  moving  from  one 
envelope  to  another;  they  have  the  appropriate  weapon  ready  for  use  when 
entering  the  envelope.  They  have  a  lower  probability  of  flying  into 
terrain  or  losing  an  aircraft  due  to  fuel  exhaustion. 

Throttle  activity  was  deemed  the  most  significant  situation  awareness 
variable  contained  in  the  existing  data  base. 

Control  inputs.  Useful  information  about  the  level  of  ACM  skill  can 
be  derived  by  examining  the  way  the  pilot  flies  the  aircraft.  A  highly 
skilled  pilot  is  able  to  fly  the  aircraft  at  the  limits  of  its  performance 
parameters  when  necessary.  It  is  important  to  note  that  this  does  not 
mean  that  the  pilot  constantly  maneuvers  the  aircraft  at  maximum  angle  of 
attack,  but  rather  is  able  to  maximize  the  tradeoff  between  turn  rate  and 
energy  expenditure  in  each  situation. 

As  previously  noted,  important  information  can  be  gleaned  from  pilot 
control  Inputs.  The  skilled  pilot  reacts  to  changing  situations  by  throt¬ 
tle  and  speedbrake  changes  but  the  less  skilled  pilot  is  likely  to  main¬ 
tain  a  relatively  high  power  setting  throughout  an  engagement,  reflecting 
either  a  lack  of  situation  awareness  or  an  overwhelming  workload. 

There  is  some  disagreement  among  experts  as  to  whether  control  smooth¬ 
ness  provides  an  indication  of  ACM  skill.  The  consensus  is  that  aileron 
smoothness  is  essential  at  high  angles  of  attack  to  prevent  loss  of  con¬ 
trol.  At  other  times,  the  Importance  of  control  smoothness  is  less 
clear. 


Application  of  basic  fighter  maneuvers.  Basic  fighter  maneuvers  (BFM) 
are  used  primarily  to  maneuver  into  the  weapons  envelope  against  a  non- 
reactlve  target.  BFM,  however,  forms  the  foundation  on  which  all  maneuvers 
in  ACM  are  based.  The  skilled  pilot  is  able  to  apply  BFM,  using  parts  of 
several  maneuvers,  in  combination  or  succession,  in  order  to  arrive  in 
firing  position. 

Candidate  measures.  Based  on  results  of  the  analytical  phase,  a  group 
of  28  candidate  measures  was  produced.  These  measures  could  be  derived 
from  time  histories  of  ACM  engagements  flown  on  the  SAAC.  The  list  included 
measures  of  engagement  outcome,  relative  aircraft  position,  control  acti¬ 
vity,  aircraft  maneuvering,  and  situation  awareness.  The  measures  were 
chosen  to  reflect  the  dimensions  of  ACM  skill  just  described.  Details  on 
their  derivation  can  be  found  in  Kelly  et  al.  (1979). 

EMPIRICAL  DATA  COLLECTION  PHASE 

After  the  analytical  phase  was  completed,  collection  of  empirical  ACM 
data  began.  Performance  and  time  history  data  were  collected  during  a  total 
of  405  engagements  on  the  SAAC  at  Luke  AFB.  The  configuration  of  the  SAAC 
was  described  in  detail  in  Kelly  et  al.  (1979)  Technical  Report. 

Pilots.  A  total  of  30  F-4  qualified  Air  Force  pilots  took  part  in 
this  portion  of  the  study.  The  pilots  were  selected  on  the  basis  of  their 
prior  training  and  ACM  experience  and  were  divided  on  this  basis  into 
three  groups  of  10  pilots  each,  representing  three  ACM  experience  level 
ranging  from  novice  to  expert. 

Procedure.  During  each  of  five  consecutive  weeks,  six  pilots  (two 
from  each  of  the  three  experience  levels)  took  part  in  the  testing.  The 
first  of  three  days  of  data  collection  each  week  was  devoted  to  carefully 
defining  the  skill  levels  of  each  participant.  After  completing  a  detailed 
questionnaire  concerning  their  flight  experience,  especially  recent  ACM 
experience,  the  pilots  completed  an  orientation  flight  in  the  SAAC.  They 
then  flew  a  set  of  pretest  exercises  involving  five  attacks  on  targets 
being  flown  in  a  predetermined  pattern  by  one  of  two  project  pilots. 

The  second  and  third  days  of  data  collection  each  week  consisted  of  a 
round  robin  series  of  free  engagements  among  the  six  pilots.  Every  pilot 
flew  against  each  of  the  other  five  pilots  three  times  in  F-4  versus  F-4 
competition.  Three  neutral  initial  setups  were  used.  On  the  third  day  of 
testing,  the  two  most  experienced  ACM  pilots  flew  six  engagements  of  dis¬ 
similar  air  combat  with  the  SAAC  configured  as  F-4  versus  MIG  21. 

Data  were  recorded  on  67  different  variables  for  each  cockpit.  These 
included  measures  of  relative  aircraft  position,  energy  states,  weapons 
swltchology  and  control  Inputs.  Project  pilot  ratings  of  the  participants 
and  peer  rankings  were  also  obtained.  From  these  67  measures  could  be 
calculated  the  28  candidate  performance  measures  described  previously. 
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Multivariate  discriminant  analyses  were  performed  on  the  data  obtained 
during  the  F-4  versus  F-4  free  engagements.  Based  on  these  analyses,  an 
algorithm  was  developed  containing  13  measures  which  accounted  for  over 
51Z  of  the  variance  in  the  performance  data  and  which  could  discriminate 
between  members  of  the  high  and  low  skill  level  groups  with  over  92% 
reclassification  accuracy. 

Examination  of  the  results  of  the  effort  suggested  an  alternative 
measurement  structure  which  might  be  used  to  account  for  a  larger  portion 
of  the  variance  and  might  provide  more  diagnostic  information.  This  struc¬ 
ture,  termed  TACSPACE,  Involved  a  multi-dimensional  state  space  model  in¬ 
volving  relative  aircraft  positions.  The  purpose  of  the  effort  reported  in 
this  document  was  to  develop  and  refine  the  TACSPACE  concept,  then  test  its 
validity,  generalizability,  and  efficacy  using  the  existing  database. 
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III.  ANALYSIS  METHOD 


The  previous  work  (Kelly  et  al.,  1979)  focused  on  the  development  of 
measurement  models  to  distinguish  novice  ACM  pilots  from  those  more 
experienced.  Two  rudimentary  models  were  created  from  measures  of  relative 
aircraft  position,  energy  states,  weapons  switchology,  and  control  inputs. 
The  first  model,  referred  to  as  Whole  Engagement  Success  Factors,  related 
to  the  use  of  aircraft  controls  and  gross  measures  of  relative  positional 
superiority,  and  can  be  called  a  technique  model.  The  second  model,  refer¬ 
red  to  as  the  TACSPACE  Model,  related  entirely  to  the  relative  positions 
of  the  opponents  and  was,  therefore,  a  tactics  model. 

The  Kelly  et  al.,  (1979)  study  represented  the  development  and  prelim¬ 
inary  analysis  of  data  collected  during  ACM  free  engagements.  The  measure¬ 
ment  structures  thus  developed  contained  the  framework  for  an  effective 
ACM  performance  measurement  system.  While  the  whole  engagement  technique 
model  was  excellent  for  the  classification  of  pilots  into  skill  levels, 
performance  diagnosis  was  difficult  using  this  approach.  It  was  suggested 
that  a  combination  of  the  two  models  might  provide  the  needed  capability 
by  imposing  a  context-specific  structure  on  the  control  activity  of  the 
pilot.  The  following  sections  describe  the  further  development  and  refi¬ 
nement  of  an  ACM  performance  measurement  approach. 

While  whole  engagement  data  analysis  could  successfully  classify 
pilots  into  skill  levels,  little  diagnostic  information  was  found  by  this 
approach.  Summary  data  confirmed  that  experienced  pilots  tend  to  adjust 
their  throttle  more,  roll  and  pitch  more  adroitly,  use  the  vertical  plane 
to  their  advantage,  and  keep  their  opponent  in  view.  Quantifying  the  ten¬ 
dency  to  do  these  things  is  different  from  telling  someone  when  and  where 
inappropriate  control  activity  took  place  during  the  course  of  an  engage¬ 
ment.  A  different  measurement  structure  was  needed  to  provide  diagnostic 
capability. 

The  relative  position  and  orientation  of  the  opposing  aircraft  are  of 
great  importance,  since  the  main  goal  of  an  engagement  is  to  achieve  a 
position  for  weapons  release.  Therefore,  diagnosis,  at  a  minimum  must  be 
done  in  the  context  of  relative  aircraft  states. 

During  training  in  ACM,  pilots  are  taught  maneuvering  rules  for 
various  combinations  of  range,  range  rate,  line-of-sight  angle,  and 
aspect  angle  between  their  aircraft  and  the  target  aircraft.  Line-of- 
sight  (LOS)  is  the  angle  of  the  opponent's  aircraft  off  the  aircraft  axis 
or  nose  of  the  defender.  The  aspect  angle  used  for  the  defender  is  the 
angle  off  the  opponent  aircraft  axis  or  tail  of  the  defender.  This  angu¬ 
lar  relationship  is  diagramed  in  Figure  3. 


Given  that  range,  LOS,  and  aspect  angle  sufficiently  describe  the 
situation,  performance  assesssment  can  be  based  on  control  activity  as  a 
function  of  these  three  states.  This  is  done  by  creating  a  coordinate 
system  called  TACSPACE,  using  range,  LOS,  and  aspect  as  the  axes.  TACSPACE 
can  be  broken  into  a  matrix  of  sectors  or  cells  (Figure  4).  The  cells  of 
TACSPACE  reflect  tactical  function  or  expected  population  densities  and 
are  not  necessarily  of  equal  size. 

In  the  previous  work,  TACSPACE  was  divided  into  125  cells  along  these 
three  dimensions  (LOS  angle,  aspect  angle  and  range).  A  preliminary  ana¬ 
lysis  of  a  very  small  sample  of  cells  examined  the  feasibility  of  this 
approach. 

It  was  found  that  the  percentage  of  time  spent  in  certain  cells  of 
TACSPACE  tended  to  discriminate  between  winners  and  losers.  Also,  it  was 
found  that  pilots  occupying  a  given  cell  flew  differently  depending  on 
their  skill  level.  These  results  suggested  that  further  work  with  TACSPACE 
should  provide  a  more  descriptive  and  diagnostic  measurement  model  than  is 
represented  by  the  Whole  Engagement  algorithm.  The  new  approach  is  illus¬ 
trated  in  Figure  5  and  is  described  in  the  following  paragraphs. 

SEGMENT  ENGAGEMENTS 

Expert  pilots  sometimes  make  errors  during  engagements  resulting  in 
unnecessary  losses.  Conversely,  relatively  novice  pilots  occasionally  win 
an  engagement  through  accident  or  circumstance.  Obviously  these  uncharac¬ 
teristic  performances  contribute  a  certain  amount  of  error  variance  to  the 
analysis.  Presorting  the  data  according  to  engagement  outcome  as  well  as 
level  of  training  is  not  the  final  solution.  It  is  possible  for  a  pilot 
to  gain  and  lose  the  advantage  more  than  once  within  an  engagement,  yet 
ultimately  win. 

There  are  three  apparent  conditions  that  occur  during  an  engagement: 
(a)  a  totally  neutral  situation  when  neither  pilot  has  the  advantage,  (b) 
an  offensive  situation  when  the  defender  is  approaching  a  weapon  release 
envelope,  and  (c)  a  defensive  situation  which  is  the  direct  opposite  of 
the  offensive  situation.  An  expert's  ability  to  gain  the  advantage  from 
the  neutral  situation,  maintain  the  offensive  and  escape  from  the  defen¬ 
sive  positions  are  of  greatest  interest. 

Therefore,  to  preserve  the  terminal  status  of  each  segment,  it  was 
necessary  to  divide  the  time  history  into  segments  and  sort  into  groups  of 
trained  winners  against  relatively  untrained  losers.  When  not  considering 
the  entire  engagement,  positions  of  relative  advantage,  disadvantage  or 
neutrality  were  defined  in  terms  of  llne-of-sight,  aspect  angle  and  range. 
Three  primary  contrasting  situations  were  created: 


Figure  5.  Flow  Diagram.  Analysis  Methods. 
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1.  Initially  neutral  and  emerging  in  either  a  defensive  or  offensive 
relative  position. 

2.  Initially  offensive  and  terminating  either  by  firing  or  losing  the 
offensive. 

3.  Initially  defensive  and  terminating  either  by  being  killed  or  by 
escaping. 

The  definition  of  offensive  and  defensive  regions  were  highly  depen¬ 
dent  on  the  weapons  selected  by  each  of  the  two  pilots.  Unfortunately,  it 
was  impossible  to  uniquely  Identify  weapons  selection  from  the  switches 
monitored  in  the  existing  data  base.  For  analysis  purposes,  it  was  assumed 
that  the  ideal  weapons  selection  strategy  was  followed  during  the  course 
of  the  engagement  by  both  pilots.  The  segment  boundaries  related  to  the 
most  extreme  angles  and  ranges  allowed  for  either  missiles  or  guns. 

PERFORM  TAC SPACE  CALCULATIONS 

Two  sets  of  performance  measures  were  used.  The  set  called  time- 
history  segment  summary  measures,  listed  in  Table  1,  were  applied  to  time- 
history  segments.  If  these  measures  were  applied  to  the  entire  engagement, 
the  result  would  be  comparable  to  the  Whole  Engagement  Success  Factors 
which  had  been  developed  previously  (Kelly  et  al.,  1979).  A  different  set 
of  measures,  called  TACSPACE  cell  summary  measures,  listed  in  Table  2, 
were  applied  to  performance  within  a  single  TACSPACE  cell.  While  there  is 
one  time-history  segment  summary  measure  set  associated  within  any  par¬ 
ticular  time-history  segment,  there  may  be  a  number  of  TACSPACE  cell 
measure  sets,  one  relating  to  each  TACSPACE  cell  occupied  during  that 
time-history  segment.  The  resulting  summary  measure  sets  can  each  be  used 
to  develop  a  discriminant  model  for  performance  measurement.  For  the  con¬ 
venience  of  the  user  the  analysis  software  generated  four  transforms  for 
each  selected  variable.  Coincidentally,  the  two  candidate  measure  sets 
are  quite  similar  in  this  particular  case. 

The  previous  exploration  of  the  TACSPACE  approach  was  very  limited  and 
was  performed  manually.  Software  development  was  therefore  required  to 
perform  the  analyses  of  this  study  in  a  timely  fashion.  New  data  base 
architecture,  graphic  display  programs  and  file  manipulation  programs  were 
implemented  to  allow  the  TACSPACE  manipulation  and  discriminant  analysis 
algorithm  to  work  jointly  and  effectively. 

Dynamic  graphic  display  routines  were  developed  to  evaluate  engagement 
segmentation,  to  verify  calculation  of  cell  measurement  transforms,  and  to 
more  rapidly  identify  practical  TACSPACE  regions  or  significant  patterns 
of  activity  in  TACSPACE.  These  time  history  plots  of  performance  measures 
or  TACSPACE  activity  were  invaluable  for  rapidly  debugging  the  new  analysis 
routines  for  effective  presentation  of  analysis  results.  They  were  also 
useful  for  checking  for  proper  segmentation  of  the  time  history  data. 


19 


i'J.BLE  1.  CANDIDATE  TIME -HISTORY  SEGMENT  SUMMARY  MEASURES 


Number  Measure  Units 

1  AVG  Airspeed  Knts 

2  RNG  Airspeed  Knts 

3  RMS  Airspeed  Knts 

4  AVG  Turn  Rate  Rad/Sec 

5  RNG  Turn  Rate  Rad/Sec 

6  RMS  Turn  Rate  Rad/Sec 

7  AVG  G  -  Forces  - 

8  RNG  of  G  -  Forces 

9  RMS  G  -  Forces 

10  AVG  LOS  Azimuth  Deg 

11  LOS  Azimuth  Range  Deg 

12  RMS  LOS  Azimuth  Deg 

13  AVG  Closing  Rate  Ft/Sec 

14  RNG  Closing  Rate  Ft/Sec 

15  RMS  Closing  Rate  Ft/Sec 

16  AVG  Throttle  Position  - 

17  RNG  Throttle  Position 

18  RMS  Throttle  Position  - 

19  AVG  Roll  Rate  Deg/Sec 

20  RNG  Roll  Rate  Deg/Sec 

21  RMS  Roll  Rate  Deg/Sec 

22  AVG  Plane  of  Action  Deg 

23  RNG  Plane  of  Action  Deg 

24  RMS  Plane  of  Action  Deg 

25  AVG  LOS  Elevation  X  Neg 

26  AVG  %  Aspect  +  LOS  >  180 

27  AVG  %  Aspect  +  LOS  <  180 

28  AVG  Lateral  Velocity  Ft/Sec 

29  RNG  Lateral  Velocity  Ft/Sec 

30  RMS  Lateral  Velocity  Ft/Sec 


AVG  *  Average 
AAV  **  Average  Absolute 
RMS  ■  Root-Mean  Squared 
SD  =  Standard  Deviation 
RNG  *  Range 


TABLE  2.  CANDIDATE  TACSPACE  CELL  SUMMARY  MEASURES 


Number 

Measure 

Units 

1 

AVG  Airspeed 

Knts 

2 

AAV  Airspeed 

Knts 

3 

RMS  Airspeed 

Knts 

4 

SD  Airspeed 

Knts 

5 

AVG  Turn  Rate 

Rad /Sec 

6 

AAV  Turn  Rate 

Rad /Sec 

7 

RMS  Turn  Rate 

Rad/Sec 

8 

SD  Turn  Rate 

Rad /Sec 

9 

AVG  G  -  Force 

- 

10 

AAV  G  -  Force 

- 

11 

RMS  G  -  Force 

- 

12 

SD  G  -  Force 

- 

13 

AVG  LOS  Azimuth 

Rad 

14 

AAV  LOS  Azimuth 

Rad 

15 

RMS  LOS  Azimuth 

Rad 

16 

SD  LOS  Azimuth 

Rad 

17 

AVG  Closing  Rate 

Ft/Sec 

18 

AAV  Closing  Rate 

Ft/Sec 

19 

RMS  Closing  Rate 

Ft /Sec 

20 

SD  Closing  Rate 

Ft/Sec 

21 

AVG  Throttle  Position 

- 

22 

AAV  Throttle  Position 

- 

23 

RMS  Throttle  Position 

- 

24 

SD  Throttle  Position 

- 

25 

AVG  Roll  Rate 

Rad /Sec 

26 

AAV  Roll  Rate 

Rad /Sec 

27 

RMS  Roll  Rate 

Rad/Sec 

28 

SD  Roll  Rate 

Rad /Sec 

29 

AVG  Plane  of  Action 

Deg 

30 

AAV  Plane  of  Action 

Deg 

31 

RMS  Plane  of  Action 

Deg 

32 

AVG  Plane  of  Action 

Deg 

AVG  =  Ave  rage 
AAV  *  Average  Absolute 
RMS  =  Root-Mean  Squared 
SD  =  Standard  Deviation 
RNG  -  Range 


Programs  were  implemented  to  perform  the  cell  merger,  TACSPACE  re¬ 
dimensioning,  and  cell  measure  transformations  required  for  the  analysis. 
A  program  was  also  written  to  interface  the  TACSPACE  analysis  with  the 
already  existing  discriminant  analysis  programs. 

NONPARAMETRIC  DISCRIMINANT  ANALYSIS 

Some  difficulty  was  encountered  in  applying  standard  multivariate 
discriminant  analyses  with  the  TACSPACE  approach.  Not  every  pilot  was 
observed  in  any  specific  segment  of  TACSPACE,  and  an  even  smaller  subset 
was  observed  in  a  particular  cell  in  that  segment.  Therefore,  these  ana¬ 
lyses  can  often  be  characterized  by  a  large  number  of  variables  and  a  rela¬ 
tively  small  number  of  observations.  If  an  attempt  was  made  to  use 
standard  discriminant  analysis  methods,  the  degrees  of  freedom  would 
vanish  or  the  mathematics  would  yield  a  trivial  solution.  Nonparametric 
techniques  were  therefore  used  in  these  situations. 

The  t-test  and  Tukey  Quick  Test  of  Location  (TQTL)  (see  Appendix  A) 
were  applied  to  the  TACSPACE  cell  occupancy  data  and  summary  measures. 
The  TQTL  was  found  to  have  critical  limitations  when  applied  to  the 
TACSPACE  data.  The  t-test  became  the  only  metric  for  measure  selection 
in  the  nonparametric  analyses.  The  t-test  was  used  to  select  measures, 
and  the  unit-scaling  procedure  (see  Appendix  A)  was  used  to  provide 
weighting  coefficients.  Nonparametric  models  were  devised  for  each  set  of 
measures,  and  the  results  were  used  to  guide  more  elaborate  multivariate 
discriminant  analyses  for  the  more  suitable  data  sets. 

SUITABILITY  DECISIONS 

Once  TACSPACE  was  defined  and  the  resulting  discriminant  model  calcu¬ 
lated,  statistical  indicators  and  graphic  representations  could  be  used  to 
determine  whether  the  time-history  segmentation  and  cell  structure  was 
suitable.  The  segmentation  and  TACSPACE  definition  was  then  modified  to 
produce  the  desired  balance  between  complexity  and  discriminant  quality. 

The  appropriate  discriminant  analyses  were  performed  on  the  summary 
measures  to  produce  assessment  models  relating  to  the  entire  segment  or 
the  individual  TACSPACE  cells.  If  these  results  proved  to  be  inadequate, 
the  entire  process  was  repeated  from  whatever  starting  point  seemed 
appropriate,  until  no  further  improvements  could  be  produced. 
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IV.  ANALYSIS  RESULTS 


Many  measurement  models  were  developed.  The  models  varied  greatly  in 
complexity  and  ability  to  account  for  performance  variance.  The  final 
models  are  described  in  detail  in  Appendix  B,  Functional  Specifications, 
and  the  coefficients  are  tabulated  in  Appendix  C,  Data  for  Discriminant 
Models.  Appendix  D  contains  normative  data. 

This  section  contains  the  results  of  the  (a)  similar  aircraft  analy¬ 
sis,  (b)  reduced  measure  set  analysis  for  the  ACMI/TACTS,  (c)  dissimilar 
aircraft  analysis  and  (d)  weapons  envelope  study.  The  similar  aircraft 
analysis  is  outlined  in  Table  3,  along  with  a  reference  to  the  appropriate 
figure  or  table  number  to  assist  the  reader.  Table  numbers  beginning  with 
"C-"  are  located  in  Appendix  C.  Notice  that  the  TACSPACE  structure,  or 
resolution,  changed  as  a  result  of  the  initial  whole  engagement  analyses, 
and  that  all  remaining  analyses  used  the  medium  resolution  TACSPACE  struc¬ 
ture. 

Three  types  of  analyses  were  performed.  First,  the  TACSPACE  analysis 
contained  only  measures  of  the  percent  time  that  the  pilots  occupied  each 
cell  in  TACSPACE.  Second,  the  cell  summary  analysis  contained  the  perfor¬ 
mance  measures  within  each  TACSPACE  cell;  these  measures  were  shown  in 
Table  2,  Section  III.  Third,  the  segment  summary  analysis  contained  per¬ 
formance  measures  taken  from  the  beginning  of  a  segment  (neutral,  offen¬ 
sive  or  defensive)  to  the  end,  independent  of  the  TACSPACE  cells  through 
which  the  aircraft  may  have  passed;  these  measures  were  shown  in  Table  1, 
Section  III.  Generally,  the  unit-scaled,  non-pa rame trie  discriminant  ana¬ 
lysis  was  used,  but  there  were  a  few  cases  where  it  was  possible  to  per¬ 
form  a  parameteric,  ridge-adjusted  multiple  discriminant  analysis;  these 
exceptions  are  noted  in  the  text. 

WHOLE  ENGAGEMENT  ANALYSIS,  SIMILAR  AIRCRAFT 

Entire  engagements  were  analyzed.  A  time  history  subfile,  exclusively 
composed  of  engagements  between  novices  and  experts,  was  generated.  This 
subfile  required  4  hours  of  CPU  time  to  build,  and  it  occupied  25  percent 
of  the  allocated  115  thousand  block  space  on  the  VAX  computer.  This  pro¬ 
vided  a  benchmark  of  the  largest  possible  volume  of  data;  also,  it  pro¬ 
duced  results  that  were  directly  comparable  to  the  prior  analyses.  As 
expected,  the  same  pattern  of  winners  and  loosers  shown  in  Kelly  et  al. 
(1979)  was  observed,  but  differences  in  the  performance  measurement  models 
resulted. 

TACSPACE  -  high  resolution  analysis.  TACSPACE  processing  was  started 
with  an  11  by  8  by  7  cell  structure  defined  in  Figure  6.  Out  of  the  616 
TACSPACE  cells,  499  were  found  to  be  occupied  by  one  of  the  groups  during 
the  engagement.  Table  4  shows  the  unit-scaled  discriminant  model  of  these 
cells.  This  model,  even  in  its  simplicity,  accounted  for  more  than  50 
percent  of  the  ACM  maneuvering  variance. 
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TABLE  3.  SIMILAR  AIRCRAFT  ANALYSES 


TACSPACE  Resolution 

Figure  or 

Analysis 

High 

Medium 

Low 

Table  Number 

Whole  Engagement 

TACSPACE,  Hi  Resolution 

X 

Figure  6,  Table  A 

Relative  Advantage 

X 

Table  5 

TACSPACE,  Low  Resolution 

X 

Figure  7 

Cell  Summary 

X 

Table  6 

TACSPACE,  Med.  Resolution 

X 

Figure  8,  Table  C-2 

Segment  Summary 

N/A 

Table  C-l 

Neutral  Segment 

TACSPACE 

X 

Table  C-A 

Cell  Summary 

X 

Table  7 

Segment  Summary 

N/A 

Table  C-3 

Offensive  Segment 

TACSPACE 

X 

Table  C-6 

Cell  Summary 

X 

Table  8 

Segment  Summary 

N/A 

Table  C-5 

Defensive  Segment 

TACSPACE 

X 

Table  C-8 

Cell  Summary 

X 

Table  9 

Segment  Summary 

N/A 

Table  C-7 

Summary  of  Similar  Analysis 


Table  10 


E 


TABLE  4.  WHOLE  ENGAGEMENT  TACSPACE  DISCRIMINANT  VARIABLE 


Mean 

S.D. 

Novices 

8.324 

9.460 

Experts 

-12.131 

9.811 

Pooled 

-  1.903 

14.054 

Break  Point:  -2.265 

FI:  264.6  F2:  30833.6 

Wilkes  -  Lambda:  0.468  S-OMEGA:  0.528 
0.531  Estimated  Error:  15.38% 


Relative  advantage  and  disadvantage.  An  overall  measure  of  tactical 
maneuvering  skill  was  evident  in  the  TACSPACE  computer  plots  for  entire 
engagements.  A  plot  of  contrasts  between  novice  and  expert  groups  in 
terms  of  LOS  and  aspect  can  be  seen  in  Table  5.  The  greater  percentage  of 
time  that  the  novices  spent  in  each  LOS/aspect  cell,  independent  of  range, 
is  shown  in  this  table.  As  the  sum  of  LOS  and  aspect  increases,  the 
novice  is  more  likely  to  spend  more  time  in  the  corresponding  cells  than 
is  the  expert.  Of  course,  a  sum  equal  to  360  degrees  represents  the  con¬ 
dition  of  greatest  disadvantage.  There  is  a  clear  line  of  demarcation; 
this  line  represents  the  condition  where  the  Bum  is  equal  to  180  degrees. 
Consequently,  the  percentage  of  time  the  sum  of  LOS  and  aspect  is  greater 
than  180  degrees  can  be  considered  as  a  measure  of  relative  disadvantage. 

The  statistical  importance  of  relative  disadvantage  during  the  whole 
engagement  can  be  implied  from  Table  C-l ,  Measure  10;  relative  disadvan¬ 
tage  has  the  highest  communality  (.714)  of  the  twelve  measures  in  the 
model.  This  model,  however,  accounted  for  only  37  percent  of  the  variance. 
More  striking  is  the  importance  of  relative  advantage  in  the  neutral 
segment  multivariate  discriminant  model  (Table  C-3,  Measure  27);  its 
communality  (.989)  is  an  order  of  magnitude  higher  than  the  other  mea¬ 
sures,  and  the  neutral  segment  model  accounted  for  91  percent  of  the 
variance. 

TACSPACE  -  low  resolution  analysis.  It  became  clear  that  many  cells 
in  TACSPACE  were  occupied  infrequently  or  for  short  durations.  Little 
information  can  be  acquired  from  analysis  of  such  cells  by  themselves. 
Contiguous  cells  of  low  information  content  or  highly  similar  information 
content  can  be  combined.  This  facilitates  the  analysis  and  future  appli¬ 
cation  of  the  results  by  reducing  the  number  of  cells  in  TACSPACE  and  by 
finding  those  cells  which  might  acquire  more  meaning  by  their  merger. 
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TABLE  5.  CONTRAST  TABLE  OF  LOS  AND  ASPECT  FOR  WHOLE  ENGAGEMENT. 

LOS 


1  2 

3 

A 

5 

6 

7 

8 

A 

1 

_ 

1 

S 

2 

+ 

3 

6 

P 

3 

+ 

1 

10 

8 

E 

A 

1 

20 

23 

11 

C 

5 

+ 

3 

19 

30 

26 

1A 

T 

6 

• 

1 

10 

2A 

35 

28 

17 

7 

+  + 

1 

8 

22 

33 

26 

27 

^Percentage  of  engagement  x  10  that  less  experienced  pilots  occupied 
TACSPACE  cells  more  than  expert  pilots.  Symbols  represent  data  less  than 
one-tenth  of  a  percent  as  follows: 

.1  >  +  >  .05 
.05  >  -  >  .02 
.02  >  .  >  0 


Three  criteria  were  used  to  combine  cells:  (a)  contiguous  cells  which 
had  not  been  occupied,  (b)  contiguous  cells  which  had  been  occupied  about 
the  same  amount  of  time  by  both  groups  being  compared,  and  (c)  contiguous 
cells  where  behavior  was  apparently  influenced  by  weapon  capabilities. 
Great  care  had  to  be  exercised  when  using  the  second  two  criteria  so  that 
measures  of  pilot  performance  were  not  confounded  or  deleted  by  improper 
merging  of  cells. 

A  reduction  of  TACSPACE  to  a  low  resolution  version  was  performed  to 
determine  the  magnitude  of  information  that  would  be  lost  by  merging 
cells.  In  the  interest  of  time  and  software  complexity,  the  cells  were 
merged  by  combining  entire  rows  or  columns  of  cells.  This  was  done  by 
dropping  breakpoints  along  individual  TACSPACE  axes,  based  on  the  dis¬ 
tribution  of  cell  occupancy  data  for  the  whole  engagement.  The  resulting 
5  by  A  by  A  TACSPACE  is  shown  in  Figure  7. 

This  80-cell  TACSPACE  dramatically  reduced  the  analysis  time  and  stor¬ 
age  space  requirements.  Another  unit-scaled  discriminant  model  was  created 
using  69  of  the  cells  and  accounted  for  A6  percent  of  the  variance  between 
the  two  groups.  This  was  a  considerable  reduction  of  model  complexity 
(from  A99  to  69  measures)  with  a  loss  of  only  A  percent  of  the  descriptive 
power.  This  interim  analysis  is  not  presented  in  Appendix  C  because  a 
subsequent  analysis  improved  this  model. 
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Cell  summary  analysis.  Nonparametric  discriminant  models  were  deve¬ 
loped  using  various  cells  from  the  existing  low-resolution  TACSPACE.  The 
cell  coordinates  (in  terms  of  range,  LOS  and  aspect),  the  number  of  obser¬ 
vations  in  each  group,  the  number  of  significant  measures  used  in  the 
model,  and  the  resulting  r2  values  can  be  found  in  Table  6. 


TABLE  6.  LOW  RESOLUTION  WHOLE  ENGAGEMENT 
TACSPACE  CELL  SUMMARY  MODELS 


Range , 

Cell 

LOS. 

Aspect 

No.  of 

Novice 

Observations 

Expert 

No.  of  Measures 
Used  in  Model 

R2^ 

1 

1 

1 

8 

35 

14 

.15 

2 

1 

1 

11 

41 

16 

.15 

3 

1 

1 

9 

37 

15 

.29 

5 

1 

1 

12 

30 

7 

.22 

1 

2 

1 

16 

55 

18 

.17 

2 

2 

1 

19 

60 

11 

.15 

3 

3 

1 

40 

83 

16 

.17 

2 

1 

1 

24 

66 

18 

.14 

2 

2 

l 

43 

88 

19 

.17 

3 

2 

1 

48 

82 

17 

.15 

2 

3 

l 

75 

81 

19 

.15 

3 

3 

1 

77 

85 

17 

.16 

3 

4 

1 

84 

36 

20 

.24 

2 

2 

l 

78 

81 

19 

.27 

3 

3 

1 

54 

13 

16 

.23 

3 

1 

1 

23 

61 

20 

.14 

4 

1 

1 

12 

41 

15 

.25 

Examination  of  the  results  of  this  cross  section  of  TACSPACE  shows 
that  only  limited  Information  was  derived  from  the  discriminant  models. 
In  a  majority  of  the  cases,  the  measurement  functions  described  less  than 
18  percent  of  pilot  performance.  All  of  the  models  accounted  for  only  a 
trivial  portion  of  the  total  variance  compared  to  what  normally  would  be 
expected. 

This  disappointing  result  may  be  explained  in  any  of  several  ways. 
First,  the  data  were  sorted  by  skill  level  and  not  by  engagement  outcome. 
Performances  which  ultimately  led  to  victory  were  confounded  with  those 
that  resulted  in  defeat.  Even  though  the  general  trend  was  for  experts  to 
win  more  engagements,  it  was  possible  that  a  delicate  relationship  was 
nullified  by  a  few  defeats.  Second,  when  the  cells  were  merged,  some 
effects  may  have  been  lost.  Third,  TACSPACE  is  still  large  (80  cells)  and 
a  limited  number  of  engagements  were  used  in  this  comparison.  Considering 
the  variability  of  performance  that  could  be  expected  in  each  cell,  the 
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limited  number  of  observations  could  detract  from  finding  any  significant 
relationship.  Fourth,  the  reduction  in  resolution  was  performed  without 
considering  its  effect  on  the  contained  measures.  The  resulting  sample 
size  and  complexity  of  the  model  become  limiting  factors.  Some  rearrange¬ 
ment  of  cell  boundaries  might  produce  more  significant  models. 

TACSPACE  -  medium  resolution  analysis.  One  corrective  action  was  to 
break  LOS  and  aspect  into  positive  and  negative  regions  based  on  the  sign 
(plus  or  minus)  of  the  elevation  of  those  two  angles.  Maneuvering  right 
and  left  usually  results  in  symmetrical  performance.  Unfortunately,  this 
is  not  true  for  performance  in  the  vertical  axis  of  the  aircraft.  Also, 
the  maneuvering  behavior  may  be  different,  depending  on  whether  the  oppo¬ 
nent  is  above  the  wing  line  (positive  LOS  and  probably  in  sight)  or  below 
the  wing  line  (negative  LOS  and  possibly  not  in  sight)  of  the  aircraft. 
The  opponent's  view  was  expressed  by  a  negative  elevation  of  aspect  angle. 
Note  that  the  sign  does  not  relate  to  left  or  right,  only  to  up  or  down. 

The  medium  resolution  (and  final)  TACSPACE  structure  is  illustrated  in 
Figure  8.  Whole  engagement  TACSPACE  was  re-analyzed  using  this  structure, 
as  shown  in  Table  C-2.  The  new  structure  accounted  for  54  percent  of  the 
variance,  which  represented  an  improvement  of  8  percent  over  the  low  reso¬ 
lution  model.  Thus,  all  remaining  analyses  (neutral  segment,  offensive 
segment,  defensive  segment,  dissimilar  aircraft  and  reduced  measure  set 
for  ACMI/TACTS)  were  performed  with  positive  and  negative  LOS  and  aspect 
elevation  angles. 

Segment  summary  analysis.  There  were  no  segments  in  the  whole  engage¬ 
ment  analyses.  Twenty-four  time-history  summary  measures  for  whole  enga¬ 
gement  data  produced  a  unit-scaled  discriminant  model  (15  measures)  that 
accounted  for  about  30  percent  of  the  variance.  The  same  measures  were 
re-analyzed  using  the  parametric  multivariate  techniques  to  produce  a 
discriminant  model  accounting  for  about  37  percent  of  the  variance. 

The  time-history  segment  summary  measures,  composed  of  aircraft  per¬ 
formance  and  maneuvering  variables,  provided  relatively  weaker  results 
when  compared  to  TACSPACE  and  to  previous  summary  models.  When  these  sum¬ 
mary  models  were  examined,  the  critical  roles  of  situation  awareness  and 
relative  advantage  measures  were  apparent.  The  pilot  performance  is  taken 
out  of  the  specific  positional  context  in  an  analysis  of  an  entire  engage¬ 
ment.  Thus,  only  general  control  differences  show  up  in  these  models.  In 
general,  these  measures  alone  may  not  be  a  powerful  delineator  of  skills. 
These  analyses,  however,  led  to  the  full  set  of  30  time-history  segment 
summary  measures  used  in  the  rest  of  this  study.  The  re-analysis  of  the 
new  measure  set  for  the  whole  engagement  accounted  for  more  than  63  per¬ 
cent  of  the  variance,  as  shown  in  Table  C-l. 


-180  -120  -60  -20  0  20  60  120  180 

LOS* 


*Negative  LOS  means  line-of-sight  below  the  wing  line.  Negative  Aspect 
means  the  angle  is  below  the  wing  line  of  the  opponent. 


Figure  8.  Final  TACSPACE  Model. 
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NEUTRAL  SEGMENT  ANALYSES,  SIMILAR  AIRCRAFT 

The  time  history  data  were  segmented  into  those  portions  of  the  enga¬ 
gement  where  neither  aircraft  was  within  an  opponent's  weapons  envelope. 
The  data  from  each  segment  were  divided  into  groups  based  on  the  pilots' 
relative  situation  at  the  end  of  the  segment.  The  data  from  the  cockpits 
terminating  in  a  defensive  or  disadvantaged  relative  position  went  into 
one  group.  The  data  from  those  terminating  in  an  offensive  or  advanta¬ 
geous  position  went  into  the  other.  The  placement  of  data  into  groups  was 
additionally  constrained  in  that  only  the  skilled  or  expert  pilot  data 
were  permitted  into  the  offensive  group.  This  allowed  a  certain  control 
over  the  quality  of  the  criterion  data  by  allowing  inclusion  of  only  the 
superior  performance  data. 

TACSPACE  analysis.  A  unit-scaled  discriminant  model  of  the  TACSPACE 
transformation  of  these  data  accounted  for  84  percent  of  the  neutral 
maneuvering  performance  variance  (Table  C-4).  The  results  are  similar  to 
those  obtained  with  the  relative  advantage  measure  analysis,  but  on  a 
smaller  scale.  Starting  from  a  neutral  condition,  the  more  skilled  pilots 
consistently  gravitated  to  specific  regions  in  the  direction  of  improved 
relative  advantage. 

Cell  summary  analysis.  Neutral  TACSPACE  contained  a  large  number  of 
cells.  A  complete  model  including  all  possible  TACSPACE  cell  summary 
equations  would  be  prohibitively  complex  for  application  In  the  FAAC  or 
ACMI/TACTS.  As  a  check  on  the  previous  poor  performance  of  cell  summary 
models,  analyses  were  performed  on  several  neutral  cells.  Table  7  sum¬ 
marizes  the  results  of  seven  representative  TACSPACE  summary  discriminant 
analyses.  Looking  back  at  the  whole  engagement  summary  measures  in  Table 
4,  it  appears  that  the  new  segmenting,  sorting  and  TACSPACE  definition  had 
beneficial  effects  on  these  models.  On  the  average,  the  new  cell  summary 
models  accounted  for  more  of  the  variance,  although  less  than  half  of  them 
are  significant.  Better  models  resulted  from  those  cells  that  had  the 
highest  number  of  occurrences  in  common  with  both  groups. 


TABLE  7.  NEUTRAL  SEGMENT  TACSPACE  CELL  SUMMARY  MEASURES 


Cell 

Range,  LOS,  Aspect 

16  6 

1  7  6 

2  7  6 

3  7  6 

4  7  6 

5  7  6 

3  8  6 


No.  of  Observations 
Defensive  Offensive 


35 

4 

19 

15 

31 

36 

24 

21 

11 

11 

7 

11 

4 

58 

No.  of  Measures 
Used  in  Model  R2 


14 

.225 

16 

.255 

16 

.315 

11 

.347 

7 

.223 

14 

.464 

16 

.156 
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Segment  summary  analysis.  The  unit-scaled  time-history  segment  sum¬ 
mary  model  accounted  for  over  67  percent  of  the  general  performance  and  of 
the  tendency  to  improve  one's  relative  advantage.  As  a  demonstration  of 
possible  gains  with  the  advanced  multivariate  techniques,  a  ridge  discrim¬ 
inant  model  was  developed.  This  model  accounted  for  more  than  91  percent 
of  the  same  between-groups  variance  (Table  C-3).  Since  past  experience 
has  shown  typical  gains  of  10  to  15  percent  over  the  nonparametric  tech¬ 
niques,  this  is  an  exceptional  case. 

OFFENSIVE  AND  DEFENSIVE  TACSPACE  ANALYSES,  SIMILAR  AIRCRAFT 

Two  more  segments  of  time  history  data  were  developed  in  the  same 
fashion  as  was  done  for  the  neutral  zone  analysis.  An  offensive  segment 
was  produced  to  study  the  difference  between  those  who  lost  or  kept  their 
advantage  by  dividing  the  offensive  time-history  data  into  groups  by  the 
outcome  of  the  segment.  Likewise,  the  time  histories  of  those  on  the 
defensive  were  divided  into  those  who  escaped  or  those  who  fell  into  a 
weapons  envelope  recognized  by  the  opponent  (i.e.,  were  fired  upon).  The 
same  TACSPACE  processing  was  done  on  these  two  sets  of  groups. 

TACSPACE  analysis.  The  unit-scaled  TACSPACE  model  for  the  offensive 
segment  accounted  for  about  24  percent  of  the  variance  (Table  C-6).  The 
defensive  model  (Table  C-8)  accounted  for  more  than  45  percent.  More 
detailed  descriptions  of  these  models  may  be  found  in  Appendix  C. 

Cell  summary  analysis.  The  discriminant  models  of  TACSPACE  cell  sum¬ 
mary  measures  provided  encouraging  results.  Tables  8  and  9  summarize  the 
results  of  several  analyses.  The  number  of  defensive  and  offensive  cells 
occupied  by  both  groups  was  relatively  small.  The  number  of  observations 
in  each  cell  also  was  small.  The  tables  reveal  that  once  pilots  entered 
defensive  cells  they  were  more  likely  to  be  killed  than  to  escape.  Con¬ 
versely,  once  an  offensive  stance  is  assumed,  the  opponent  is  not  likely 
to  be  lost.  Again,  this  is  in  line  with  the  fact  that  skilled  pilots 
dominated  the  advantageous  relative  positions. 

Even  though  some  of  the  sample  sizes  are  small,  the  offensive  and 
defensive  cell  summary  models  are  accounting  for  more  variance  (by  several 
orders  of  magnitude)  than  the  time-history  segment  summary  models.  This 
is  a  very  strong  argument  for  context  specific  measurement  of  pilot  per¬ 
formance  in  the  offensive  and  defensive  situations. 
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TABLE 

8.  OFFENSIVE 

SEGMENT  TACSPACE  CELL 

SUMMARY  MEASURES 

l 

X 

Cell 

No.  of 

Observations 

No.  of  Measures 

-  .4 

* 

Range 

LOS, 

Aspect 

Escape 

In  Envelope 

Used  in  Model 

R2 

5 

5 

5 

3 

10 

23 

.324 

'  1 

4 

6 

5 

3 

9 

16 

.665 

''  1 

4 

5 

5 

6 

14 

12 

.250 

9 

3 

6 

5 

10 

13 

8 

.247 

-  * 

• 

3 

5 

5 

12 

24 

8 

.152 

■  1 

2 

6 

5 

12 

16 

19 

.262 

j 

2 

5 

5 

19 

20 

19 

.302 

1 

2 

4 

5 

2 

7 

17 

.494 

.  ] 

1 

6 

5 

6 

10 

10 

.584 

J 

1 

1 

5 

5 

8 

12 

22 

.625 

• 

1 

4 

5 

4 

3 

12 

.878 

J 
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4 

•  ■ 

; 

TABLE 

9.  DEFENSIVE  SEGMENT  TACSPACE  CELL 

SUMMARY  MEASURES 

•  ' 

X 

Cell 

No.  of 

Observations 

No.  of  Measures 

•  1 

Range 

,  LOS. 

Aspect 

Escape 

In  Envelope 

Used  in  Model 

R2 

■  ■ 

1 

8 

1 

8 

11 

16 

.259 

.  •  j 

■«< 

■ 

4 

8 

8 

4 

10 

16 

.636 

_ 

3 

8 

8 

6 

11 

16 

.546 

• 

1 

8 

8 

5 

8 

21 

.629 

-  •  i 

2 

8 

8 

16 

13 

18 

.453 

4 

8 

7 

2 

13 

20 

.197 

3 

8 

7 

10 

17 

13 

.204 

2 

8 

7 

17 

19 

13 

.247 

’  • 

1 

8 

7 

7 

12 

5 

.133 

•  ' 

* 

1 

8 

2 

9 

5 

15 

.457 

-  •  j 

5 

8 

1 

3 

6 

17 

.597 

4 

8 

1 

5 

10 

15 

.583 

3 

8 

1 

8 

15 

9 

.268 

2 

8 

1 

11 

15 

9 

.268 

1 

8 

i 

8 

11 

16 

.259 

• 

-i 

• 

• 
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Segment  summary  analysis.  Both  offensive  and  defensive  time-history 
segment  summary  models  accounted  for  insignificant  amounts  of  performance. 
These  data  (Tables  C-5  and  C-7)  are  provided  for  documentation  rather  than 
as  useful  material  for  implementation. 

SUMMARY  OF  SIMILAR  AIRCRAFT 

The  prominent  similar  aircraft  analysis  results  are  summarized  in 
Table  10.  Several  conclusions  are  offered:  The  TACSPACE  cell  summary 
measures  describe  pilot  control  behavior  in  the  tail  chase;  they  are  less 
descriptive  of  tactical  maneuvering.  Once  an  offensive  position  is 
attained,  it  requires  more  anticipation  and  rigorous  control  activity  than 
tactics  to  bring  the  opponent  into  a  weapons  envelope.  Escape  has  more  to 
do  with  the  pilot's  relative  position  and  control  performance  than  does 
making  a  kill.  Although  these  conclusions  may  be  obvious  to  ACM  pilots, 
it  is  Important  to  note  that  the  performance  measurement  models  were  able 
to  isolated  them* 


TABLE  10.  SUMMARY  OF  SIMILAR  AIRCRAFT  ANALYSES 


R2 

Measures 

No.  of  Models 

Whole  Engagement 

Segment  Summary 

.63 

13 

1 

TACSPACE 

.46 

69 

1 

Cell  Summary 

.15-. 29 

7-16 

17 

Neutral  Segment 

Segment  Summary 

.91 

9 

1 

TACSPACE 

.84 

128 

1 

Cell  Summary 

.16-. 46 

7-16 

7 

Defensive  Segment 

Segment  Summary 

.05 

12 

1 

TACSPACE 

.45 

7 

1 

Cell  Summary 

.13-. 63 

5-21 

15 

Offensive  Segment 

Segment  Summary 

.02 

14 

1 

TACSPACE 

.24 

11 

1 

Cell  Summary 

.21-. 83 

9-15 

11 
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REDUCED  MEASURE  SET  ANALYSIS  FOR  ACMI /TACTS 


One  of  the  major  criteria  established  for  the  ACM  performance  measure¬ 
ment  system  is  that  it  must  be  applicable  on  a  variety  of  training  devices 
rather  than  being  SAAC  specific.  An  environment  in  which  a  measurement 
system  would  provide  an  important  tool  is  the  ACMI  TACTS. 


In  the  strictest  sense,  the  ACMI/TACTS  is  a  simulator  of  weapons  deli¬ 
very.  The  system  electronically  tracks  up  to  eight  aircraft  and,  through 
telemetry,  is  able  to  compute  precisely  all  interrelationships  between 
these  aircraft.  When  one  combatant  launches  a  simulated  missile  or  gunshot, 
the  ground-based  computers  calculate  the  path  of  the  ordnance  and,  after 
adding  a  factor  for  the  potential  unreliability  of  the  weapon  system, 
indicate  whether  the  target  aircraft  was  killed. 


To  make  these  calculations,  the  ACMI/TACTS  must  collect  and  store 
huge  amounts  of  data  concerning  aircraft  position,  states  and  switchology. 
Currently  this  data  pool  is  primarily  used  to  provide  a  playback  capabi¬ 
lity  during  debriefing.  The  only  kind  of  performance  measurement  actually 
made  is  the  mission-level  score  of  number  of  kills.  However,  several 
groups  are  currently  working  to  develop  the  advanced  capabilities  of  this 
system.  In  the  future,  the  ACMI/TACTS  may  become  an  important  source  of 
ACM  performance  measurement  data  as  well  as  being  the  excellent  testbed 
for  aircraft  and  tactics  which  it  now  represents. 


A  potential  difficulty  exists  for  the  direct  transfer  of  the  SAAC  ACM 
performance  measurement  package  to  the  ACMI/TACTS  environment  because  the 
ACMI/TACTS  is  incapable  of  measuring  stick  and  throttle  movements.  However, 
re-analyses  were  made  of  most  of  the  relevant  time-history  segment  summary 
measures  and  cell  summary  measures  without  throttle  and  plane-of-action 
measures.  The  neutral  model  is  presented  in  Table  C-3.  Tables  11  and  12 
summarize  the  defensive  and  offensive  cell  summary  models. 


TABLE  11. 


DEFENSIVE  SEGMENT  REDUCED  TACSPACE  CELL 
SUMMARY  MEASURES  SET 


Range , 

Cell 

LOS, 

Aspect 

No.  of 
Escape 

Observations 

In  Envelope 

No.  of  Measures 

Used  in  Model  R2 

4 

8 

1 

5 

10 

11 

.585 

3 

8 

1 

8 

15 

11 

.325 

2 

8 

1 

11 

15 

7 

.229 

2 

8 

7 

17 

19 

8 

.221 

2 

8 

8 

16 

13 

12 

.339 

TABLE  12.  OFFENSIVE  SEGMENT  REDUCED  TACSPACE  CELL 
SUMMARY  MEASURES 


Range , 

Cell 

LOS, 

Aspect 

No.  of 
Escape 

Observations 

In  Envelope 

No.  of  Measures 
Used  in  Model 

1 

4 

5 

4 

3 

9 

.831 

1 

5 

5 

8 

12 

15 

.684 

1 

6 

5 

6 

10 

9 

.524 

2 

5 

5 

19 

20 

15 

.212 

DISSIMILAR  AIRCRAFT  ANALYSIS 

It  was  possible  to  perform  only  two  major  analyses  on  the  dissimilar 
aircraft  data  due  to  the  very  limited  number  of  these  engagements  col¬ 
lected.  Segmentation  and  cell  summary  analyses  could  not  be  performed 
with  just  a  few  observations  in  each  group,  therefore,  all  analyses  were 
performed  on  the  whole  engagement. 

TACSPACE  analysis.  The  unit-scaled  TACSPACE  model  for  the  offensive 
segment  accounted  for  about  84  percent  of  the  variance  (C-10).  This 
result  compares  favorably  to  the  similar  aircraft,  although  there  is  an 
explicit  tactical  maneuvering  difference  between  the  two  conditions.  This 
difference  is  clearly  seen  when  the  occupancy  plots  are  compared  and  is 
also  reflected  in  the  cells  that  comprise  each  model. 

Segment  summary  analysis.  The  reduced,  ACMI/TACTS  measure  set  was 
analysed  for  the  whole  engagement,  and  accounted  for  over  66  percent  of 
the  variance  in  a  unit-scaled  model  (C-9). 

Some  information  was  sacrificed  by  the  removal  of  throttle  and  plane- 
of-action  measures,  but  the  result  is  not  so  severe  as  to  render  them 
significantly  less  useful  than  models  which  could  be  implemented  on  SAAC. 
Application  of  the  TACSPACE  and  summary  models  should  benefit  both  simula¬ 
tor  and  aircraft  environments. 

WEAPON  ENVELOPE  STUDY 

Accurate  recognition  of  the  weapon  envelope  is  an  important  factor  in 
the  successful  completion  of  an  engagement.  Good  tactical  skills  cannot 
overcome  the  inability  to  unleash  weapons  within  their  effective  limits. 
The  measures  in  Table  13  were  developed  to  study  the  state  of  the  aircraft 
when  the  pilots  thought  they  were  in  the  envelope.  Although  it  was  impos¬ 
sible  to  track  weapons  selection  throughout  this  engagement,  the  nature 
and  condition  of  the  trigger  pull  could  be  determined.  This  was  done  by 
matching  each  trigger  pull  in  the  time  history  data  to  the  weapon  indi¬ 
cated  in  the  real  time  engagement  printout  saved  during  data  collection. 
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Essentially,  the  computer  made  the  first  guess  and  the  researcher  manually 
Inserted  the  correct  weapon  from  the  engagement  printout  for  each  of  sev¬ 
eral  hundred  trigger  pulls.  Thus,  each  time  a  pilot  pulled  the  trigger, 
the  kind  of  weapon  selected  and  the  indicated  measures  were  recorded  for 
analysis. 


TABLE  13.  KILL  ENVELOPE  ANALYSIS  VARIABLES 


Variable  No.  Name 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 
17 


Range 

LOS 

Aspect 

Closing  Rate 

LOS  Rate 

Aspect  Rate 

Delta  Turn  Rate 

Delta  Plane  of  Action 

In  Envelope 

Kill 

Absolute  Closing  Rate 
Absolute  LOS  Rate 
Absolute  Aspect  Rate 
Absolute  Delta  Turn  Rate 
Absolute  Plane  of  Action 
Dummy  Start  1 
Dummy  Start  2 


It  should  not  be  possible  to  fire  a  low-aspect  AIM-9  missile  and 
achieve  a  kill  when  the  opponent  aircraft  is  approaching.  However  a 
number  of  such  kills  did  occur  in  the  data  collected.  An  investigation 
revealed  that  it  was  indeed  possible  in  the  simulator.  It  was  also  dis¬ 
covered  that  whoever  fired  first  would  lock-out  the  trigger  of  the  oppo¬ 
nent  for  the  duration  of  firing.  These  two  anomalies  surely  had  an  Impact 
on  the  study  of  kill  envelopes  as  well  as  the  course  of  the  engagements. 


Three  other  realistic  weapons  situations  were  studied  in  detail. 
Trigger-pull  events  were  divided  into:  (a)  high-aspect  guns,  (b)  low- 
aspect  guns,  and  (c)  low-aspect  missiles.  Discriminant  models  were  built 
based  on  relative  experience  level.  The  results  of  these  analyses  are 
summarized  in  Table  14.  There  was  very  little  difference  between  novice 
and  expert  pilots  as  to  when  they  were  likely  to  pull  the  trigger.  The 
experienced  pilots  had  more  kills  than  the  novice  pilots,  apparently 
because  they  had  gained  more  opportunities  to  shoot,  not  because  of  supe¬ 
rior  marksmanship. 
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TABLE  14.  KILL  ENVELOPE  STUDY  RESULTS 


Relative  Position 

High  Aspect 
Low  Aspect 
Low  Aspect 


Weapon  r£ 

Gun  0.124 

Gun  0.192 

Aim9  0.139 
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V.  DEVELOPMENT  OF  FUNCTIONAL  SPECIFICATIONS 


One  study  objective  was  to  use  the  analyses  from  this  study  to  provide 
functional  specifications  for  a  performance  measurement  system  that  will 
serve  to  aid  training  in  the  SAAC.  To  this  end,  current  training  was 
observed  to  determine  the  information  needed  by  the  IPs  and  the  information 
not  provided  by  current  instrumentation.  Then,  SAAC  operations  and  support 
personnel  were  interviewed  to  determine  constraints  on  implementation  of 
additional  measurements  and  displays.  Finally,  functional  specifications 
were  developed  which  should  be  suitable  for  design  and  development  of  an 
add-on  display  and  measurement  system. 

CURRENT  TRAINING 

The  SAAC  is  currently  used  primarily  to  train  F-4  ACM  pilots  in  the 
TAC  ACES  program.  The  TAC  ACES  training  syllabus  contains  the  following 
set  of  simulator  sessions: 

1.  Familiarization 

2 .  Weapons  Employment 

3.  BFM  -  Offensive 

4.  BFM  -  Offensive/Counter  Offensive 

5.  BFM  -  Counter-Offensive/Low  Altitude 

6 .  Advanced  BFM 

7.  Introduction  to  Threat,  Introduction  to  2vl  (two  against  one) 

8.  ACM  (2vl) 

9.  MIG  (Russian  design  aircraft)  Exploitation 

10.  Turkey  Shoot 

In  addition,  the  SAAC  is  used  for  special  training.  For  example,  training 
sessions  were  observed  in  which  reconnaissance  pilots  were  taught  to  be 
credible  adversaries  so  that  they  could  teach  others  in  their  units  how  to 
defend  themselves. 

For  the  most  part,  the  IP  occupied  the  second  cockpit  while  teaching. 
Consequently,  most  of  the  information  needed  by  the  IP  was  acquired  from 
the  cockpit  instruments  and  a  visual  assessment  of  the  relative  spatial 
positions  of  the  simulated  aircraft.  Also,  heavy  use  was  made  of  problem 
freeze.  The  IP  would  move  through  the  engagement  in  short  increments,  stop- 


ping  to  instruct  and  critique  at  critical  moments.  Consequently,  when  the 
IP  needed  information  in  a  fast-moving  situation,  he  could  freeze  the 
action  to  make  a  careful  assessment. 

During  those  training  sessions  where  both  cockpits  were  occupied  by 
students  (e.g.,  2vl),  the  IP  used  the  Instructor  Station.  His  primary  need 
there  was  a  graphical  display  of  the  relative  positions  of  the  two  air¬ 
craft.  This  display  is  currently  provided  at  the  console. 

There  is  no  method,  however,  for  providing  a  global  score  of  perfor¬ 
mance,  either  during  a  session,  or  at  the  end  of  the  session.  Also,  there 
are  no  means  for  providing  diagnostic  end-of-run  measurement  to  indicate 
performance  relative  to  others  who  have  performed  in  the  same  training. 
It  is  concluded,  therefore,  that  functional  specifications  should  emphasize 
measurement  and  display  of  such  summary  and  comparative  information. 

The  investigators  observed  that  significant  changes  have  been  made  in 
TAC  ACES  training  since  the  original  data  were  obtained.  In  contrast  to  a 
training  program  which  largely  involved  trial -and-error  experimentation  by 
the  students,  a  rigidly  structured  program  consisting  of  demonstration  and 
practice  has  evolved.  As  a  result  only  the  Turkey  Shoot  at  the  end  of  the 
syllabus  Is  comparable  to  the  conditions  under  which  the  data  of  this  study 
were  collected.  Consequently,  recommendations  for  measurement  based  on 
this  study  apply  only  to  this  phase  of  training. 

IMPLEMENTATION  CONSTRAINTS 

Engineering  and  operations  personnel  associated  with  the  SAAC  were 
interviewed  to  determine  limitations  for  add-on  measurement  and  display. 
No  clear-cut  limits  could  be  established.  There  is  a  large  capability  for 
additional  measurement,  but  a  specific  proposal  would  have  to  be  generated 
to  obtain  a  firm  assessment  of  feasibility.  The  following  comments  were 
noted  to  guide  the  development  of  a  feasible  specification: 

1.  A  Sigma  5/7  computer  with  64K  memory  is  available  for  measurement 
computation.  Extensive  computations  could  be  accomplished  in  a 
short  time  directly  after  each  flight. 

2.  Even  extreme  estimates  of  the  amount  of  measurement  computation, 
with  sampling  rates  of  2  Hz,  seemed  to  be  within  capability. 

3.  The  ADAGE  graphical  display  is  quite  full;  however,  a  simple  dis¬ 
play  may  be  possible  in  addition  to  the  current  display.  Communica¬ 
tion  speed  with  the  ADAGE  will  probably  be  a  severe  limitation. 

4.  Additional  cockpit  display  would  be  limited  to  use  of  an  existing 
cockpit  instrument  (e.g.,  tachometer)  which  is  not  used  fir  SAAC 
flights. 


41 


DESIGN  APPROACH  ADOPTED 

Based  on  the  foregoing  analyses,  a  decision  was  made  to  compute  three 
different  scores:  (a)  relative  advantage  (percentage  of  time  the  sum  of 
LOS  and  aspect  Is  greater  than  180  degrees),  (b)  a  discriminant  score 
based  on  summary  measures,  which  can  be  termed  a  technique  discriminant 
score,  and  (c)  a  discriminant  score  based  on  TACSPACE,  which  can  be  termed 
a  tactics  discriminant  score.  These  data  would  be  displayed  directly 
after  each  session,  along  with  a  tabulation  of  the  summary  measures.  An 
optional  addition  is  a  real-time  display  of  one  of  the  previous  three 
scores,  selected  by  the  IP,  computed  and  displayed  on  a  thermometer-type 
display  as  the  engagement  progresses. 

The  analyses  of  this  study  showed  that  LOS  and  aspect,  independent  of 
range,  allowed  the  computation  of  a  measure  of  relative  advantage  which 
was  a  strong  discriminator  between  novices  and  experts.  Since  this  score 
is  both  easy  to  compute  and  easy  to  interpret,  it  is  recommended  that  this 
score  be  displayed. 

The  measurement  analyses  performed  in  this  study  result  in  the  selec¬ 
tion  of  a  set  of  measures  that  collectively  discriminate,  as  well  as  the 
definition  of  a  score  that  can  be  used  for  discrimination  between  experts 
and  novices.  Furthermore,  a  discriminant  score  can  also  be  computed  using 
nonparametric  techniques  and  TACSPACE.  The  first  discriminant  score  is 
based  on  measures  of  aircraft  control  and  can  be  termed  a  technique  dis¬ 
criminant  score;  the  second  is  based  on  measures  of  relative  position  in 
TACSPACE  and  can  be  termed  a  tactics  discriminant  score.  The  discriminant 
function  scores  can  be  computed  during  each  training  flight  for  display 
directly  after  the  training  session.  In  this  way,  overall  scores  will  be 
available  for  unbiased  comparison,  together  with  the  set  of  summary 
measures.  It  is  proposed  that  normative  data  be  collected  so  that  stu¬ 
dents  can  readily  compare  their  performance  with  that  of  others. 

The  analyses  performed  in  this  study  include  development  of  discrimi¬ 
nant  models  for  the  neutral,  offensive  and  defensive  portions  of  TACSPACE; 
consequently,  one  could  consider  display  of  technique  and  tactics  discri¬ 
minant  scores  for  each  of  three  zones  as  well  as  the  total  engagement. 
However,  from  examination  of  the  variance  accounted  for,  only  the  neutral 
zone  scores  are  strong  candidates.  Better  discrimination  could  be  obtained 
by  using  a  model  which  changes  with  each  TACSPACE  cell;  however,  it  is 
believed  that  such  an  implementation  is  infeasible.  It  is  concluded  from 
these  considerations  that  only  neutral  zone  and  total  engagement  discrimi¬ 
nant  scores  should  be  computed. 

Furthermore,  as  an  option,  depending  on  the  feasibility  of  display  on 
the  ADAGE,  one  of  the  discriminant  scores  (as  selected  by  the  IP)  will  be 
computed  in  real  time  during  the  training  flight  so  that  a  continuous 
assessment  of  performance  is  available.  A  simple  thermometer-type  scale 
in  the  upper  left  corner  of  the  ADAGE  (one  display  for  each  student)  is 
proposed  for  the  optional  display  of  the  progressive  discriminant  score. 


•  4 


•  4 


•  1 
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The  format  for  the  thermometer-type  display  is  shown  in  Figure  9, 
and  the  display  format  for  summary  and  discriminant  scores  is  shown  in 
Figure  10. 

The  specifics  of  the  proposed  add-on  measurement  and  display  are  pre¬ 
sented  in  Appendix  B  in  a  form  which  should  support  development  of  the 
necessary  software.  No  hardware  engineering  changes  are  believed  to  be 
necessary. 


Positioned  to 
Indicate  Score 


Positioned  to 
Indicate  Break¬ 
point  between 
Novices  and 
Experts 


Figure  9.  Real-Time  Display  Format. 
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X 
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b:  normative  data  previously  collected 

Figure  10.  Summary  Display  Format. 
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VI.  CONCLUSIONS 


As  a  result  of  this  effort,  it  was  concluded  that: 

1.  SAAC  provides  an  excellent  environment  for  the  Implementation  of 
an  Instructor  oriented  system  for  ACMPM. 

2.  There  was  little  difference  between  skilled  and  unskilled  ACM 
pilots  in  their  ability  to  recognize  effective  weapons  envelopes.  The 
primary  difference  between  the  two  groups  of  pilots  was  in  their  ability 
to  maneuver  into  the  weapons  firing  envelope. 

3.  The  TACSPACE  concept  provides  an  excellent  framework  on  which  to 
build  an  ACMPM  system;  however,  additional  intelligence  should  be  added  to 
have  a  comprehensive  system. 

4.  A  relatively  simple  matrix  involving  the  sum  of  aspect  and  LOS 
angles  provides  a  powerful  "how-goes-it"  indicator  of  maneuvering  during 
an  engagement  that  is  conducted  with  limited  aspect  weapons.  This  offers 
the  potential  for  a  useful  interim  metric. 

5.  The  training  procedures  used  on  the  SAAC  have  changed  since  the 
data  used  for  this  analysis  were  collected.  The  syllabus  of  instruction 
and  instructional  methods  have  evolved  to  the  point  that  the  database  from 
the  program  is  applicable  only  to  the  last  phase  of  training,  the  "Turkey 
Shoot . " 


6.  Because  very  little  active  instruction  is  done  from  the  console, 
and  the  engagements  are  frequently  halted  using  "Freeze,”  the  utility  of  a 
real-time  performance  measurement  display  on  the  console  has  to  be  seri¬ 
ously  questioned.  Nevertheless,  several  display  concepts  were  developed; 
they  can  provide  performance  information  to  the  instructor  at  the  console 
(or  the  instructor  in  the  cockpit)  during  applicable  phases  of  training. 
It  is  recommended,  however,  that,  for  future  applications  in  the  SAAC, 
emphasis  be  placed  on  performance  measurement  to  be  used  for  post-training 
debriefing,  rather  than  on  real-time  performance  measurement. 

7.  The  measurement  algorithms  could  be  adapted  to  the  more  restricted 
measurement  capabilities  of  the  "Basic"  ACMI/TACTS  with  only  a  slight  loss 
of  efficacy.  They  could  be  adapted  to  a  more  complex  ACMI/TACTS,  such  as 
the  system  at  Nellis  AFB,  with  little  or  no  loss  of  efficacy.  Because  they 
measure  free  engagements  (rather  than  an  active  and  interruped  instruc¬ 
tional  process),  they  might  be  better  suited  for  ACMI/TACTS  application 
than  for  use  in  the  SAAC  at  the  present  time.  Because  of  possible  set-up 
differences  and  other  than  1  versus  1  engagements,  new  data  should  be 
collected  from  the  ACMI/TACTS  and  analyzed  before  implementation. 


8.  There  were  instances  of  questionable  data  in  the  database  used  for 
this  analysis,  primarily  due  to  software  peculiarities  in  the  SAAC  at 
the  time  these  data  were  collected.  Many  of  the  noted  problems  have 
since  been  corrected.  Also,  as  noted  previously,  the  results  of  this 
analysis  have  limited  applicability  for  future  implementation  because  of 
changes  in  instructional  methods.  For  these  reasons,  a  new  set  of  data 
should  be  collected  and  analyzed  before  implementation  of  these  ACMPM 
concepts.  It  should  be  noted  that  the  data  analysis  methods  have  matured 
to  a  point  where  a  future  analysis  could  be  performed  with  optimum 
efficiency. 

9.  The  TACSPACE  and  data  analysis  concepts  developed  as  part  of  this 
effort  provide  a  good  method  of  assessing  overall  maneuvering  performance. 
At  this  time,  however,  the  structure  and  resulting  equations  provide  only 
indirect  performance  diagnosis.  An  analyst  who  is  familiar  with  the 
structure  and  mathematics  can  infer  performance  deficiencies  from  the 
behavior  of  individual  measures,  but  the  system  does  not  directly  tell  an 
IP  precisely  what  went  wrong,  where  it  went  wrong,  and  why.  For  direct 
diagnosis  presented  so  that  IPs  would  understand,  future  work  with  ACMPM 
should  consider  combining  TACSPACE  with  the  evolving  technology  of  expert 
systems  representations,  by  inserting  some  decision  rules  to  guide  the 
selection  of  subsequent  measurement  within  appropriate  TACSPACE  cells. 
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APPENDIX  A 

DISCUSSION  OF  STATISTICAL  PROCEDURES 


PREVIOUS  WORK 

Previous  work  (Vreuls  et  al.,  1976)  has  established  and  tested  (a)  a 
descriptive  structure,  or  model,  for  obtaining  measurement  in  man-machine 
training  and  (b)  measure  selection  methods  based  on  multivariate  statisti¬ 
cal  models  which  evaluate  the  total  set  of  measures  taken  together  and 
which  produce  weighting  coefficients.  These  coefficients  were  then  used 
to  combine  the  measures  into  a  linear  equation  which  maximally  discerns 
the  change  in  performance  due  to  training. 

However,  all  of  the  previously  used  modeling  techniques  suffer  when 
applied  to  diminished  samples  of  data.  Regardless  of  sample  size,  perfor¬ 
mance  measures  are  difficult  to  analyze.  Human  performance  data,  being 
inherently  noisy  and  non-normally  distributed,  violate  many  of  the  funda¬ 
mental  assumptions  of  multivariate  analyses.  Any  multivariate  regression 
or  discriminant  procedure  begins  to  suffer  from  a  lack  of  degrees  of  free¬ 
dom  as  the  sample  size  is  reduced.  Although  regression  mathematics  still 
function  adequately  under  these  circumstances,  interpretation  of  the 
reliability  of  the  results  becomes  muddled  even  with  ridge  adjustments. 
Discriminant  analysis  mathematics  are  such  that  the  inverse  of  the  within- 
groups  variance  matrix,  used  to  estimate  the  discriminant  coefficients, 
becomes  indeterminant  or  trivial  as  the  degrees  of  freedom  decrease.  New 
techniques  need  to  be  used  or  developed  to  probe  samples  of  data  for  sig¬ 
nificant  training  measures  and  to  build  valid,  sensitive  and  rational  per¬ 
formance  models. 

The  normative  model,  referred  to  in  previous  research  (Vreuls  et  al., 
1976)  went  a  long  way  toward  equaling  the  discriminant  model  while  using  a 
very  simple  premise.  A  performance  criterion  was  based  on  the  empirical 
distribution  of  scores  made  up  of  speculatively  weighted  system  measures 
(I.e.,  measures  of  aircraft  position  and  rates)  for  fully  proficient 
pilots.  Its  major  drawback  was  the  lack  of  control  measures  (i.e.,  stick, 
rudder,  and  throttle  measures)  included  in  the  scoring  scheme.  It  would 
be  difficult,  though,  to  speculate  as  to  the  weight  to  be  assigned  to  each 
control  measure  for  its  inclusion  in  the  score.  The  lesson  learned  from 
this  example  is  that  while  the  specific  coefficients  or  weights  for  each 
measure  are  Important,  they  are  not  nearly  as  critical  as  the  inclusion  or 
exclusion  of  measures  and  the  determination  of  reasonable  values  of  the 
model  for  performance  criteria. 

T-TEST 

It  can  also  be  surmised  that  much  simpler  measurement  analysis  techni¬ 
ques  than  multivariate  analyses  can  be  used  to  develop  satisfactory  perfor¬ 
mance  models  (Wooldridge  &  Helms,  in  press).  Measure  selection  analyses 
can  be  performed  by  univariate  parametric  techniques  (Bruning  &  Kintz, 
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1977).  Considering  each  measure  Independent  of  all  other  measures,  the 
average  value  of  each  measure  on  a  pretest  can  be  compared  to  the  average 
value  of  that  measure  on  a  posttest.  A  t-test  can  be  used  to  determine 
statistically  significant  changes  occurring  in  individual  measures  due  to 
intervening  training.  This  method  differs  little  from  the  pre-screening 
programs  used  in  previous  research  to  limit  the  vast  number  of  candidate 
measures  to  the  number  that  the  analyses  programs  could  accomodate.  The 
main  difference  is  that  the  t-test  was  the  sole  standard  for  inclusion  in 
the  model. 

The  *— test  is  one  of  the  best  known  and  most  commonly  used  method  for 
the  two-sample  location  problem  discussed  earlier,  but  it  is  unsatisfactory 
in  some  respects.  The  most  serious  drawback  is  the  need  to  assume  that  the 
populations  being  sampled  have  normal  distributions.  Human  performance 
measures  are  often  skewed  or  have  other  non-normal  characteristics.  There 
is  no  consensus  among  statisticians  as  to  how  robust  the  t-test  is  or  pre¬ 
cisely  where  the  test  breaks  down.  There  is  no  doubt  that  with  small 
samples  there  are  some  distributions  which  extensively  reduce  the  reliabi¬ 
lity  of  the  t-test. 

There  are  ways  to  transform  the  sample  to  make  it  more  normally  shaped, 
but  this  requires  yet  another  mathematical  manipulation  of  the  data.  Rather 
than  fit  the  data  to  the  technique,  perhaps  it  would  be  better  to  devise  a 
statistical  procedure  unaffected  by  the  distribution.  Non-pa rame trie  or 
distribution-free  tests  need  no  test  of  normality  and  are,  therefore,  much 
more  generally  applicable.  In  non-normal  situations,  the  t-test  can  often 
be  shown  to  be  inferior  to  some  of  the  non-pa rame trie  tests. 

TUKEY'S  QUICK  TEST  OF  LOCATION 

Tukey's  Quick  Test  of  Location  (TQTL)  was  chosen  for  implementation 
out  of  a  large  number  of  existing  non-pa rame trie  tests  for  its  simplicity. 
Unlike  most  other  non-parametric  tests,  TQTL  does  not  require  the  rank 
ordering  of  data.  The  TQTL  does  not  usually  need  statistical  tables  to 

find  the  critical  values.  It  is  sufficient  to  memorize  three  numbers  and 
perhaps  also  an  elementary  algorithm. 

Furthermore,  an  improvement  was  also  made  to  the  TQTL  algorithm 
(discussed  by  Neave,  1979).  When  the  Tukey  test  fails  to  diagnose  a  real 
difference  in  means,  it  is  often  because  of  just  one  observation  taking  a 
rather  atypical  value.  If  one  sample  contains  both  the  highest  and  the 
lowest  value,  the  TQTL  yields  inconclusive  results.  Thus  a  proce¬ 
dure  was  also  implemented  whereby  one  observation  could  be  discarded.  The 
algorithm  throws  out  the  data  point  which  results  in  the  largest  possible 
Tukey  statistic.  Of  course,  the  critical  values  are  recalculated  for  the 
unequal  populations. 
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UNIT-SCALING  TECHNIQUE 


Multivariate  techniques  offer  the  only  methods  powerful  enough  to 
capture  the  complexity  of  the  real  world  while  providing  single  metrics. 
These  metrics  are  performance  functions  composed  of  several  variables  or 
measures.  Performance  functions  are  very  useful  for  higher-level  decision 
making  or  automated  program  control ,  while  at  the  same  time  being  excel¬ 
lent  descriptive  statistics.  The  simplest  of  these  performance  models  or 
functions  take  the  linear  form: 

Y  “  blxl  +  b2x2  +  b3x3  +  •  •  •  +bnxn 

The  Y  term  is  usually  the  effect  that  the  equation  is  trying  to  enumerate, 
and  the  b  values  are  weighting  coefficients  derived  by  some  multivariate 
analyses  using  several  samples  of  n  number  of  performance  measures,  or  x 
values.  In  the  past,  coefficients  have  been  the  product  of  highly  tailored 
tailored  regression  or  discriminant  analyses.  Simply  put,  multivariate 
models  evidently  combine  information  more  effectively  than  do  humans. 
Yet,  as  discussed  before,  a  noticeable  degradation  of  performance  or  loss 
of  utility  occurs  when  these  complex  techniques  are  applied  to  small 
samples. 

The  term  unit  scaling  is  used  to  describe  a  model  using  the  unit 
weighting  technique  with  full  excursion  scaling  as  opposed  to  unit  normal 
scaling.  The  predictor  variables  are  scaled  by  their  sample  range  after 
they  have  been  translated  by  the  subtraction  of  the  sample  minimum. 
Thence,  the  predictor  variables  are  relocated  as  well  as  scaled.  Although 
not  mathematically  required,  the  translation  is  performed  for  convenience. 
The  resulting  model  values  for  n  variances  will  lie  between  zero  and  n. 
For  a  set  of  n  raw  measures  the  unit  scaling  equation  would  take  the 
form: 


Y 


a2X2~^Min2 
range ^  range2 


+aSxS:™2n 

rangen 


where  a  is  still  the  unit  direction  variable,  Min  is  the  sample  minimum 
and  range  is  the  sample  maximum  minus  the  minimum. 


The  sample  location  provides  no  useful  information,  so  that  for  either 
unit  weighting  or  unit  scaling,  the  predictor  variable  could  be  relocated 
in  the  same  manner.  Disregarding  the  sign  variable  and  the  relocation, 
the  comparable  coefficients  become  respectively.  The  primary  difference 


sigma  n  range  n 

between  the  standard  deviation  and  the  range  is  that  the  latter  is  totally 
unaffected  by  the  shape  of  the  sample  distribution.  This  feature  should 
provide  more  latitude  in  the  application  of  the  equal  weighting  scheme. 


APPENDIX  B 
SPECIFICATIONS 


SCOPE 

The  purpose  of  this  appendix  is  to  provide  information  that  Bhould  be 
sufficient  to  implement  the  measurement  findings  of  the  reported  study. 
It  should  be  sufficient  in  the  sense  of  defining  the  mathematical  models 
and  the  desired  outputs.  However,  it  is  not  intended  to  fully  delimit  a 
design;  this  is  left  to  the  systems  and  applications  programmers. 

FLOW  DIAGRAM 

A  top-level  software  flow  diagram  is  provided  in  Figure  B-l.  During 
an  air-air  engagement,  the  measurement  software  must  loop  through  a  number 
of  computations  twice  per  second;  this  includes 

(1)  Sampling  of  parameters. 

(2)  Determination  of  the  cell  occupied  in  TACSPACE. 

(3)  Determination  of  the  zone  occupied  in  TACSPACE. 

(4)  Performing  some  transformation  computations  on  the  parameters  and, 
optionally,  computing  a  real-time  discriminant  score  and  displaying 
this  in  real  time. 

At  the  end  of  the  engagement,  there  are  more  transformations,  computation 
of  discriminant  scores  for  each  TACSPACE  zone,  and  displays  for  each  zone, 
as  well  as  for  the  total  engagement.  The  following  sections  will  provide 
additional  data  for  each  block  in  the  flow  diagram. 

PARAMETER  SAMPLING 

To  avoid  sampling  and  storing  parameters  for  an  entire  engagement,  a 
set  of  parameters  will  be  sampled  at  fixed  intervals  and  basic  computations 
made  during  the  time  between  samples.  The  list  of  parameters  required  is 

presented  in  Table  B-l.  Each  parameter  must  be  sampled  twice  per  second. 
TACSPACE  COMPUTATIONS 

Three-dimensional  TACSPACE  is  depicted  in  Figure  4,  Section  III,  and 
the  relationships  between  the  parameters  forming  the  axes  is  shown  in 
Figure  3,  Section  III.  The  cells  are  numbered  for  use  in  measurement 
calculations  as  shown  in  Table  B-2 .  Also,  note  that  TACSPACE  is  divided 
into  neutral,  offensive,  and  defensive  zones,  and  definitions  for  these 
zones  are  available  in  Table  B-2. 
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•  Compute  Average  •  Compute  AAE 

•  Compute  RMS  •  Compute  SD 


For  (Los  El.  Neg.)  and  (Los  +  ASP>  180) 
•  Compute  Percent 


For  each  Zone  and  Total: 

•  Compute  Discriminant  Score 

•  Display  Summary  Measures  &  Dlscr  Score 


Print  Measures 


Figure  B-l.  Top-level  flow  diagram. 
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TABLE  B-2.  NUMBERING  OF  TACSPACE  CELLS 


Range 


(R-l)  <  2000  <  (R-2)  <  4000  <  (R-3)  <  6000  <  (R-4)  <  8000  <  (R-5) 


Line-of-Slght  Angle 

(L-l)  <  -120  <  (L-2)  <  -60  <  (L-3)  <  -20  <  (L-4)  <  0 

0  <  (L-5)  <  20  <  (L-6)  <  60  <  (L-7)  <  120  <  (L-8) 


Aspect  Angle 

(A-l)  <  -170  <  (A-2)  <  -120  <  (A-3)  <  -60  <  (A-4)  <  0 

0  <  (A-5)  <  60  <  (A-6)  <  120  <  (A-7)  <  170)  <  (A-8) 


Numbering  of  TACSPACE  Cells 

Cell  Number  =  [R  +  5(L-1)  +  40  (A-l)] 

Zones 

Neutral  60  <  |  LOS  |  <  120  (L-8) 

60  <  |  ASP  j  <  120  (a-6) 

Offensive  |  LOS  |  <  60 

j  ASP  |  <  60 

|  LOS  J  >  120 
ASP  >  120 


Defensive 


A  count  of  the  number  of  times  each  TACSPACE  cell  is  occupied  is 
required  for  computing  the  nonparametric  discriminant  score.  Consequently, 
a  test  of  LOS,  Aspect,  and  Range  must  be  made  during  each  sampling  cycle, 
and  an  array  of  cell  counters  must  be  updated. 

COMPUTATIONS  DURING  EACH  SAMPLING  CYCLE 

Computations  to  be  made  during  each  0.5  second  are  listed  in  Table  B-3. 
Some  processing  is  required  for  each  parameter  and  several  tests  of  rela¬ 
tionships  are  also  necessary.  Additionally,  if  the  real-time  display 
option  is  selected  (i.e.,  proves  to  be  feasible)  additional  transformations 
of  some  parameters  and  a  weighted  sum  of  these  transformations  is  required 
during  each  sampling  cycle  (see  total  engagement  discriminant  score  for 
further  detail).  Of  course,  feasibility  will  largely  depend  on  the  availa¬ 
bility  of  sufficient  time  during  each  cycle  for  this  processing. 


TABLE  B-3.  COMPUTATIONS  TO  BE  MADE  DURING  EACH  SAMPLING  CYCLE 


EACH  PARAMETER 

•  Minimum  Value 

•  Maximum  Value 

•  Accumulative  Sum  of  Samples 

•  Accumulative  Sum  of  Samples  Squared 

ALSO 

•  Count  of  Samples  Where  LOS  El  <  0 

•  Count  of  Samples  Where  J  (LOS  +  ASP)  |  >  180 

•  Count  of  Occupation  of  TACSPACE  Zone  (N,  0,  D) 

REAL-TIME  DISPLAY  OPTION 


•  Weighted  Sum  of  Specific  Transformed  Parameters 
to  Yield  Discriminant  Score  at  Each  Sample 

Note  that  end-of-engagement  computations  will  be  made  for  each  TACSPACE 
Zone  as  well  as  for  the  total  engagment.  Therefore  the  intermediate  calcu¬ 
lations  shown  in  Table  B-3  must  be  accumulated  for  the  zone  occupied  at  the 
time  of  the  sampling  cycle  and  for  the  total  engagement.  Consequently  four 
areas  of  storage  are  required.  Note  also  that  at  this  time  available  data 
only  support  computation  of  scores  for  the  neutral  zone  and  total  engagment, 
but  provision  should  be  made  for  future  computation  of  offensive  and  defen¬ 
sive  zone  scores. 


'  • 
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END-OF-ENGAGEMENT  COMPUTATIONS 


At  the  end  of  the  engagment,  the  transformations  defined  in  Table  B-4 
must  be  computed.  Discriminant  scores  for  the  neutral  zone  and  the  total 
engagement  are  calculated  in  accordance  with  the  tables  in  Appendix  C.  As 
shown,  there  are  two  methods  for  computing  the  discriminant  score  (see 
Table  B-5).  It  should  be  apparent  from  the  form  of  the  data  which  method 
of  computation  should  be  used. 

DISPLAYS 

There  are  two  types  of  displays:  the  Summary  Display  (including  the 
discriminant  score  at  the  bottom)  as  shown  in  Figure  10,  Section  V,  and  the 
Real-Time  Display  as  shown  in  Figure  9,  Section  V. 

The  Summary  Display  Format  includes  data  corresponding  to  TACSPACE 
zones:  neutral,  offensive,  defensive,  and  total  engagement.  These  displays 
are  only  generated  at  the  end  of  the  engagement,  and  are  produced  on  any 
available  output  device  (e.g.,  line  printer).  These  displays  Include  nor¬ 
mative  data  for  comparison  to  trainee  performance.  The  trainee's  score  can 
be  compared  to  average  novice  and  expert  performance.  The  current  nor¬ 
mative  data  are  presented  in  Appendix  D.  Ideally,  provision  will  be  made 
to  update  these  numbers  after  each  scored  simulator  engagement.  The  rela¬ 
tive  advantage  score  is  based  on  computations  of  LOS  +  ASP;  the  techniques 
score  is  a  discriminant  score  based  on  the  parameters  of  Table  B-l;  and  the 
tactic  score  is  a  discriminant  score  based  on  TACSPACE  cell  counts. 

The  Real-Time  Display  is  a  thermometer-type  of  a  discriminant  score 
which  is  progressively  generated  throughout  the  engagement.  One  index  pro¬ 
vides  the  discriminant  score  at  that  instant,  and  the  other  index  provides 
the  breakpoint  between  novice  and  expert  groups.  This  display,  if  com¬ 
putation  proves  feasible,  will  be  generated  in  the  upper-left  corner  of  the 
ADAGE  display  screen  for  viewing  by  the  Instructor  at  the  Instructor's 
Console.  Optionally,  if  the  Instructor  is  to  occupy  one  of  the  cockpits 
during  the  engagement,  the  score  (no  breakpoint  information)  could  be 
displayed  on  an  unused  cockpit  instrument  (e.g.,  tachometer).  Means  must 
be  provided  so  that  the  IP  can  select  between  relative  advantage,  technique 
score,  or  tactics  score  as  the  information  indicated  on  the  thermometer- 
type  display. 

Note  that  information  is  to  be  displayed  for  each  of  the  participants 
in  the  ACM  engagement,  and  there  are  two  Real-Time  Displays  and  two  Summary 
Displays. 
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TABLE  B-4 .  TRANSFORMATIONS 
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TABLE  B-5.  DISCRIMINANT  SCORE  COMPUTATIONS 


PARAMETRIC 


Y  -  bxXx  +  b2X2  + . +  bjjXflj 

where  Xm  *  the  m^  performance  score 


NONPARAMETRIC 


Y  - 


1 

M 


Xm  ~  MIN  (Xm) 
Range 

01 


where  Xm  •  occupancy  count  of  mth  TACSPACE  cell 


APPENDIX  C 


DATA  FOR  DISCRIMINANT  MODELS 


TABLE  C-l.  MULTIVARIATE  DISCRIMINANT  SEGMENT  SUMMARY  MODEL 

FOR  WHOLE  ENGAGEMENT 


Measure 

Coefficient 

Communality 

1 

.001 

.159 

2 

-  .013 

.395 

6 

-  .549 

.154 

8 

-  .571 

.281 

10 

-  .001 

.169 

11 

.010 

.647 

12 

-  .002 

.014 

15 

.006 

.033 

16 

-1.291 

.390 

23 

2.436 

.473 

24 

-1.875 

.715 

25 

.029 

.025 

27 

-  .010 

.044 

(BP  =  9. 

3935) 

R2 

.636 

Chi2 

199.700 

ndf  1 

13.000 

ndf  2 

192.000 

F-ratio 

25.800 

60 


1 


1 


1 
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TABLE  02.  UNIT -SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT 


Measure 

1 

2 

3 

4 

5 
7 
9 

10 

15 

16 
17 
22 
23 
26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 
42 

44 

45 

47 

48 

49 

50 

53 

54 

55 

56 
59 
61 
62 
63 


Pooled  Minimum 


O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 


Pooled  Ranee 

0. 662651E-01 

1.192460 

0. 272109E-01 

2.305340 

0. 854430E-01 

2.049840 

0.328638E-01 

1.515000 

0.1732280000 

1.926380 

0. 100503E-01 

1.450320 

0.691244E-02 

1.732510 

0.388350E-01 

1.326970 

-0 . 540541E-02 

1.000000 

-0 . 196629E-01 

1.134310 

-0 . 370370E-01 

1.191370 

0.186916E-01 

1.024820 

0.231214E-01 

0.934007 

0.222222E-01 

2.647820 

0.  325087E-01 

3.218350 

0. 373832E-01 

3.900644 

0. 289855E-01 

2.996200 

0.833333E-01 

2.561460 

0.612245E-01 

2.611930 

0.1168830000 

4.949560 

0.1227440000 

5.375510 

0.2698410000 

2.797950 

0. 925926E-01 

4.153550 

0.1285710000 

1.640080 

0.1093750000 

4.967260 

0.949367E-01 

3.875140 

0.1568630000 

2.999280 

0.1851850000 

2.567570 

0.510204E-01 

2.181170 

0. 373134E-02 

1.000000 

0.787412E-01 

1.809680 

0.230415E-01 

1.230790 

0. 126263E-01 

1.362320 

0. 261194E-01 

1.173550 

0.106667E-01 

1.000000 

-0 . 270270E-02 

1 . 000000 

-0 . 699301E-02 

0.979979 

-0.209790E-01 

1.000000 

-0 . 173410E-01 

0.980201 

-0.188088E-01 

0.863446 

0. 108225E-01 

1.115250 

0.235294E-01 

2.261550 

0.203046E-01 

1.791810 
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TABLE  02.  UNIT-SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT  (Continued) 


ure 

Pooled  Minimum 

Pooled  Range 

_t 

65 

0.000000E+00 

0.228571E-01 

1.293170 

66 

O.OOOOOOE+OO 

0.272832E-01 

3.614670 

67 

0.000000E+00 

0.297297E-01 

3.134380 

68 

0.000000E+00 

0.648148E-01 

2.834160 

69 

O.OOOOOOE+OO 

0.486726E-01 

1.872840 

70 

O.OOOOOOE+OO 

0.659898E-01 

2.322070 

71 

O.OOOOOOE+OO 

0.1070500000 

4.038680 

72 

O.OOOOOOE+OO 

0.976864E-01 

5.286390 

73 

O.OOOOOOE+OO 

0.2040820000 

3.593440 

74 

O.OOOOOOE+OO 

0.609137E-01 

4.116740 

75 

O.OOOOOOE+OO 

0. 942249E-01 

3.457710 

76 

O.OOOOOOE+OO 

0.1829650000 

1.773990 

77 

O.OOOOOOE+OO 

0.1867090000 

3.755990 

78 

O.OOOOOOE+OO 

0.1992570000 

3.245790 

79 

O.OOOOOOE+OO 

0.5490200000 

1.242580 

80 

O.OOOOOOE+OO 

0.688775E-01 

2.035980 

82 

O.OOOOOOE+OO 

0 . 800000E-02 

1.295260 

86 

O.OOOOOOE+OO 

-0.271318E-01 

1.171810 

94 

O.OOOOOOE+OO 

-0.421053E-02 

1.000000 

95 

O.OOOOOOE+OO 

-0 . 133333E-01 

1.000000 

96 

O.OOOOOOE+OO 

-0.923077E-02 

1.894050 

98 

O.OOOOOOE+OO 

0.2000000000 

0.952906 

99 

O.OOOOOOE+OO 

-0.980392E-02 

1.399210 

100 

O.OOOOOOE+OO 

0.170316E-01 

0.833277 

101 

O.OOOOOOE+OO 

-0.243902E-01 

1.737460 

103 

O.OOOOOOE+OO 

0.2500000000 

0.961926 

104 

O.OOOOOOE+OO 

-0 . 230415E-02 

1.420290 

110 

O.OOOOOOE+OO 

-0.153846E-01 

1.157400 

111 

O.OOO'OOOE+OO 

-0.278552E-01 

1.013720 

113 

O.OOOOOOE+OO 

0. 342857E-01 

0.388799 

114 

O.OOOOOOE+OO 

-0.371179E-01 

0.946205 

117 

O.OOOOOOE+OO 

0.288809E-01 

1.835570 

118 

O.OOOOOOE+OO 

0. 204678E-01 

2.423290 

119 

O.OOOOOOE+OO 

0.351351E-01 

1.512430 

120 

O.OOOOOOE+OO 

0.692308E-01 

1.168450 

121 

O.OOOOOOE+OO 

-0.539084E-02 

1.897960 

123 

O.OOOOOOE+OO 

0.135136E-01 

1.523570 

124 

O.OOOOOOE+OO 

0. 135135E-01 

1.918210 

126 

O.OOOOOOE+OO 

-0 . 108303E-01 

2.090000 

128 

O.OOOOOOE+OO 

0.973236E-02 

1.000000 

129 

O.OOOOOOE+OO 

-0.520833E-02 

1 . 000000 

130 

O.OOOOOOE+OO 

-0.295699E-01 

1.421050 

132 

O.OOOOOOE+OO 

-0 . 584795E-01 

1.338130 
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TABLE  C-2 .  UNIT -SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure  Pooled  Minimum 


134 

O.OOOOOOE+OO 

136 

0.000000E+00 

137 

0.000000E+00 

138 

O.OOOOOOE+OO 

139 

O.OOOOOOE+OO 

140 

O.OOOOOOE+OO 

141 

O.OOOOOOE+OO 

142 

O.OOOOOOE+OO 

143 

O.OOOOOOE+OO 

144 

O.OOOOOOE+OO 

145 

O.OOOOOOE+OO 

146 

O.OOOOOOE+OO 

147 

O.OOOOOOE+OO 

148 

O.OOOOOOE+OO 

149 

O.OOOOOOE+OO 

150 

O.OOOOOOE+OO 

151 

O.OOOOOOE+OO 

152 

O.OOOOOOE+OO 

153 

O.OOOOOOE+OO 

154 

O.OOOOOOE+OO 

156 

O.OOOOOOE+OO 

159 

O.OOOOOOE+OO 

160 

O.OOOOOOE+OO 

162 

O.OOOOOOE+OO 

166 

O.OOOOOOE+OO 

170 

O.OOOOOOE+OO 

171 

O.OOOOOOE+OO 

172 

O.OOOOOOE+OO 

173 

O.OOOOOOE+OO 

174 

O.OOOOOOE+OO 

175 

O.OOOOOOE+OO 

176 

O.OOOOOOE+OO 

177 

O.OOOOOOE+OO 

178 

O.OOOOOOE+OO 

179 

O.OOOOOOE+OO 

180 

O.OOOOOOE+OO 

181 

O.OOOOOOE+OO 

182 

O.OOOOOOE+OO 

183 

O.OOOOOOE+OO 

184 

O.OOOOOOE+OO 

185 

O.OOOOOOE+OO 

186 

O.OOOOOOE+OO 

187 

O.OOOOOOE+OO 

188 

O.OOOOOOE+OO 

Pooled  Ranee 

t 

-0.294840E-01 

1.380090 

-0.662651E-01 

1.697400 

-0.238663E-01 

1.797220 

-0 . 1000000000 

1.954460 

-0.328638E-01 

2.224590 

-0.1732280000 

2.061410 

-0 . 144092E-01 

2.574020 

-0.481651E-01 

1.473960 

-0 . 628931E-01 

2.445190 

-0.1023620000 

1.409430 

-0.831461E-01 

1.656470 

-0.409 83 6E-01 

2.557190 

-0.1562500000 

1.516250 

-0.397112E-01 

1.636910 

-0.314961E-01 

2.563050 

-0.299539E-01 

1.465420 

-0.176471E-01 

2.406820 

-0.462963E-01 

1.003530 

-0.308483E-01 

1.732880 

-0.266667E-01 

2.119370 

-0.123457E-01 

3.063530 

-0.329670E-01 

1.812480 

0.242718E-01 

0.987333 

0. 229885E-01 

1.584690 

-0.262467E-01 

2.785560 

-0.410557E-01 

1.588110 

-0.909091E-01 

2.557680 

-0.1677220000 

2.483330 

-0.1020410000 

2.078050 

-0.404762E-01 

1.273370 

-0 . 685358E-01 

1.930070 

-0.1482650000 

3.223770 

-0.1588540000 

4.067170 

-0.1107590000 

3.994700 

-0.1012660000 

2.886000 

-0.817610E-01 

2.508030 

-0.1924290000 

3.113800 

-0.1458330000 

3.338890 

-0 . 960000E-01 

3.598130 

-0.1026600000 

3.469570 

-0.2350600000 

2.192770 

-0.2357720000 

3.846880 

-0.2514620000 

3.153610 

-0.1159870000 

3.416800 
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TABLE  C-2.  UNIT-SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

189 

O.OOOOOOE+OO 

-0.432857E-01 

4.000370 

190 

0 . 000000E+00 

-0.907080E-01 

1.930440 

191 

0.000000E+00 

-0. 69637 9E-01 

3.512450 

192 

0.000000E+00 

-0.607287E-01 

6.534390 

193 

0.000000E+00 

-0. 74074 IE-01 

6.012550 

194 

0.000000E+00 

-0.760369E-01 

5.037820 

195 

0 . 000000E+00 

-0.505319E-01 

2.934480 

199 

0.000000E+00 

0.434783E-01 

1.843870 

200 

O.OOOOOOE+OO 

0. 370370E-01 

2.044970 

201 

O.OOOOOOE+OO 

0.231959E-01 

0.916142 

202 

O.OOOOOOE+OO 

0. 306122E-01 

2.918280 

203 

O.OOOOOOE+OO 

0. 327869E-01 

1.446650 

204 

O.OOOOOOE+OO 

0.573770E-01 

2.330670 

206 

O.OOOOOOE+OO 

-0.486111E-01 

1.500170 

210 

O.OOOOOOE+OO 

0. 31791 9E-01 

0.835193 

211 

O.OOOOOOE+OO 

-0.966057E-01 

4.558730 

212 

O.OOOOOOE+OO 

-0.901639E-01 

4.703440 

213 

O.OOOOOOE+OO 

-0.1938780000 

2.398920 

214 

O.OOOOOOE+OO 

-0. 567823E-01 

1.550460 

215 

O.OOOOOOE+OO 

-0.444444E-01 

1.964600 

216 

O.OOOOOOE+OO 

-0.1263160000 

4.791010 

217 

O.OOOOOOE+OO 

-0.886076E-01 

5.919510 

218 

O.OOOOOOE+OO 

-0.1263540000 

5.498320 

219 

O.OOOOOOE+OO 

-0.370370E-01 

4.433560 

220 

O.OOOOOOE+OO 

-0.648148E-01 

3.726890 

221 

O.OOOOOOE+OO 

-0.1376520000 

4.006640 

222 

O.OOOOOOE+OO 

-0.1562500000 

4.358340 

223 

O.OOOOOOE+OO 

-0.1525420000 

3.878210 

224 

O.OOOOOOE+OO 

-0.1638842000 

2.993600 

225 

O.OOOOOOE+OO 

-0.882353E-01 

3.720680 

226 

O.OOOOOOE+OO 

-0.1463410000 

5.402070 

227 

O.OOOOOOE+OO 

-0.3089430000 

4.515350 

228 

O.OOOOOOE+OO 

-0.1971500000 

3.872550 

229 

O.OOOOOOE+OO 

-0.1500000000 

2.902900 

230 

O.OOOOOOE+OO 

-0.1445090000 

3.568100 

231 

O.OOOOOOE+OO 

-0.1365830000 

2.496220 

235 

O.OOOOOOE+OO 

0.2013420000 

1.982690 

236 

O.OOOOOOE+OO 

0.668524E-01 

1.630550 

237 

O.OOOOOOE+OO 

0.625000E-01 

5.240630 

238 

O.OOOOOOE+OO 

0.724638E-01 

6.476230 

239 

O.OOOOOOE+OO 

0. 668693E-01 

6.625570 

240 

O.OOOOOOE+OO 

0.851064E-01 

6.004000 

242 

O.OOOOOOE+OO 

0.1562500000 

1.871870 

243 

O.OOOOOOE+OO 

0. 39711 2E-01 

2.573760 
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TABLE  C-2 .  UNIT-SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

244 

0.000000E+00 

0. 393701E-01 

2.098250 

245 

O.OOOOOOE+OO 

0. 370370E-01 

3.226240 

246 

O.OOOOOOE+OO 

-0.439189E-01 

2.002400 

249 

O.OOOOOOE+OO 

0. 153061E-01 

1.278990 

250 

0.000000E+00 

0. 287611E-01 

2.120450 

251 

0.000000E+00 

-0.280374E-01 

3.777210 

252 

O.OOOOOOE+OO 

-0.470588E-01 

3.011510 

253 

O.OOOOOOE+OO 

-0.136054E-01 

2.679620 

254 

O.OOOOOOE+OO 

-0.180723E-01 

1.081240 

255 

O.OOOOOOE+OO 

-0.609137E-01 

0.864099 

256 

O.OOOOOOE+OO 

-0.590551E-01 

4.697820 

257 

O.OOOOOOE+OO 

-0 . 555556E-01 

3.995210 

258 

O.OOOOOOE+OO 

-0. 74074 IE-01 

3.583430 

259 

O.OOOOOOE+OO 

-0.856269E-01 

3.086350 

260 

O.OOOOOOE+OO 

-0.795107E-01 

2.519810 

261 

O.OOOOOOE+OO 

-0.469799E-01 

2.152660 

262 

O.OOOOOOE+OO 

-0.469799E-01 

2.195070 

263 

O.OOOOOOE+OO 

-0.641711E-01 

3.521490 

267 

O.OOOOOOE+OO 

-0.1230770000 

0.909937 

268 

O.OOOOOOE+OO 

-0.925926E-01 

1.587930 

270 

O.OOOOOOE+OO 

0.1127170000 

2.340800 

271 

O.OOOOOOE+OO 

0.2031250000 

0.913597 

272 

O.OOOOOOE+OO 

0.1812870000 

2.655800 

273 

O.OOOOOOE+OO 

0.4971750000 

4.325500 

274 

O.OOOOOOE+OO 

0.2000000000 

3.635730 

275 

O.OOOOOOE+OO 

0.3188410000 

5.237690 

276 

O.OOOOOOE+OO 

0.1013510000 

2.573460 

277 

O.OOOOOOE+OO 

0.2397660000 

4.229800 

278 

O.OOOOOOE+OO 

0.1653540000 

5.481220 

279 

O.OOOOOOE+OO 

0.1818180000 

4.021820 

280 

O.OOOOOOE+OO 

0. 774336E-01 

3.906600 

282 

O.OOOOOOE+OO 

0. 251572E-01 

1.039920 

283 

O.OOOOOOE+OO 

0.628931E-01 

2.251730 

284 

O.OOOOOOE+OO 

0. 178571E-01 

1.565400 

285 

O.OOOOOOE+OO 

0.329114E-01 

1.138990 

287 

O.OOOOOOE+OO 

-0 . 250000E-02 

1.415320 

288 

O.OOOOOOE+OO 

-0.15 00000000 

0.824839 

289 

O.OOOOOOE+OO 

0. 101266E-01 

1.723120 

290 

O.OOOOOOE+OO 

0. 505051E-01 

1.955280 

292 

O.OOOOOOE+OO 

-0 . 952381E-01 

1.414920 

293 

O.OOOOOOE+OO 

-0.203046E-01 

1.442380 

294 

O.OOOOOOE+OO 

0. 643087E-02 

1.251690 

296 

O.OOOOOOE+OO 

-0 . 173410E-01 

2.221770 

297 

O.OOOOOOE+OO 

-0.246914E-01 

1.788460 
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TABLE  C-2 .  UNIT-SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure  Pooled  Minimum 


Pooled  Range 


298 

0.000000E+00 

-0.194175E-01 

299 

0 . 000000E+00 

-0.806452E-02 

300 

O.OOOOOOE+OO 

0. 156740E-01 

306 

O.OOOOOOE+OO 

0.140602E-01 

307 

O.OOOOOOE+OO 

0.467290E-01 

308 

0.000000E+00 

0.294118E-01 

309 

O.OOOOOOE+OO 

0.496183E-01 

310 

O.OOOOOOE+OO 

0.1086960000 

311 

O.OOOOOOE+OO 

0. 358127E-01 

312 

O.OOOOOOE+OO 

0. 795455E-01 

313 

O.OOOOOOE+OO 

0.1266150000 

314 

O.OOOOOOE+OO 

0.1428570000 

315 

O.OOOOOOE+OO 

0.1111110000 

316 

O.OOOOOOE+OO 

0.993976E-01 

317 

O.OOOOOOE+OO 

0.1176470000 

318 

O.OOOOOOE+OO 

0.1417910000 

319 

O.OOOOOOE+OO 

0. 793651E-01 

320 

O.OOOOOOE+OO 

0.2350600000 

Mean 

Std  Dev 

Grpl 

4.66188 

4.67598 

Grp2 

-5.12831 

4.35224 

Pooled 

-0.23322 

6.66188 

Break  Point  ■  -1.0450 

FIs  275.996 
F2 :  32153.000 

SSB:  5607.10 

SSW:  4733.59 

SST:  10340.70 


Wilkes-Lambda :  0.457764 
S-Omega:  0.539212 
R-Squared:  0.542236 


Predicted  Min.  Percent  Error  -  15.81 


t 

1.192740 
1.565653 
1.416410 
0.961804 
1.946150 
2.016890 
0.829617 
1.268270 
3.482630 
2.874940 
3.529930 
3.423670 
3.976850 
2 . 500960 
3.972590 
3.129040 
2.727830 
2.791920 
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TABLE  C-3.  MULTIVARIATE  DISCRIMINANT  SEGMENT  SUMMARY  MODEL 

FOR  NEUTRAL  SEGMENT 
(WITH  REDUCED  ACMRI /TACTS  MEASURE  SET) 


Measure  Coefficient  Communal it 


1 

.009 

.081 

2 

.001 

.076 

3 

-  .010 

.080 

4 

•  388 

.011 

5 

-  .005 

.001 

8 

.005 

.002 

9 

.018 

.007 

25 

.095 

.009 

27 

-2.089 

.989 

R2  , 

.915 

Chiz 

1226.770 

ndf  1 

13.000 

ndf  2 

492.000 

F -Ratio 

407.7 

Simulated  Percent  Misses  16.25 
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TABLE  C-4.  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  NEUTRAL  SEGMENT 

GROUP  I  TERMINATED  DEFENSIVE  GROUP  2  TERMINATED  OFFENSIVE 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

8 

O.OOOOOOE+OO 

-0.2962960000 

1.0000000000 

9 

0 . 000000E+00 

-0.1111110000 

1 . 0000000000 

31 

O.OOOOOOE+OO 

-0.8000000000 

2 . 3860200000 

32 

O.OOOOOOE+OO 

-1 . 0000000000 

3.5964700000 

33 

0. OOOOOOE+OO 

-0 . 5000000000 

2.7625400000 

34 

O.OOOOOOE+OO 

-0.1458330000 

1.7251900000 

35 

O.OOOOOOE+OO 

-0.5000000000 

1.2138100000 

47 

O.OOOOOOE+OO 

-0.2857140000 

0.8853400000 

48 

O.OOOOOOE+OO 

-0.1111110000 

0.3813100000 

49 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

50 

O.OOOOOOE+OO 

-1.0000000000 

1.0000000000 

71 

O.OOOOOOE+OO 

-0.6842110000 

4.2756000000 

72 

O.OOOOOOE+OO 

-1.0000000000 

4.5592300000 

73 

O.OOOOOOE+OO 

-0.5714290000 

2.6475600000 

74 

O.OOOOOOE+OO 

-0.2250000000 

0.7822400000 

75 

O.OOOOOOE+OO 

-0.1875000000 

1.0000000000 

82 

O.OOOOOOE+OO 

-0. 156250E-01 

1.0000000000 

91 

O.OOOOOOE+OO 

0.1056230000 

1.3727900000 

92 

O.OOOOOOE+OO 

0.1142860000 

0.7008300000 

93 

O.OOOOOOE+OO 

-0.625000E-01 

1.0000000000 

94 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

95 

O.OOOOOOE+OO 

0.8333330000 

1.0000000000 

96 

O.OOOOOOE+OO 

0.5000000000 

1.0000000000 

97 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

98 

O.OOOOOOE+OO 

0.4074070000 

1.0000000000 

99 

O.OOOOOOE+OO 

0.1132080000 

1.4169700000 

101 

O.OOOOOOE+OO 

0.5000000000 

1.6747500000 

102 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

103 

O.OOOOOOE+OO 

0.3571430000 

1.2533300000 

104 

O.OOOOOOE+OO 

1.0000000000 

1.0364800000 

105 

O.OOOOOOE+OO 

1.0000000000 

1 . 0000000000 

106 

O.OOOOOOE+OO 

1.0000000000 

2.1143200000 

107 

O.OOOOOOE+OO 

0.6363640000 

1.1925900000 

108 

O.OOOOOOE+OO 

0.5714290000 

1.4621200000 

109 

O.OOOOOOE+OO 

0.1951220000 

1.0000000000 

110 

O.OOOOOOE+OO 

0.3333330000 

1.4167300000 

111 

O.OOOOOOE+OO 

0.1052630000 

0.5770000000 

112 

O.OOOOOOE+OO 

0.1538460000 

0.6080800000 

113 

O.OOOOOOE+OO 

0.2500000000 

0.8500800000 

114 

O.OOOOOOE+OO 

0.2195120000 

l. 0000000000 

115 

O.OOOOOOE+OO 

0.6363640000 

1.6625800000 

117 

O.OOOOOOE+OO 

-0 . 781250E-01 

0.462845E-01 

118 

O.OOOOOOE+OO 

0.1904760000 

1.0000000000 

119 

O.OOOOOOE+OO 

0.1600000000 

1 .0000000000 
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TABLE  C-4 .  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  NEUTRAL  SEGMENT  (Continued) 

GROUP  1  TERMINATED  DEFENSIVE  GROUP  2  TERMINATED  OFFENSIVE 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

127 

0.000000E+00 

-0.169492E-01 

0.574409E-01 

151 

O.OOOOOOE+OO 

0 . 2000000000 

2.0988100000 

152 

O.OOOOOOE+OO 

0.8571430000 

3.1160000000 

153 

O.OOOOOOE+OO 

1.0000000000 

2.7207600000 

154 

O.OOOOOOE+OO 

0.6000000000 

3 . 5540000000 

155 

O.OOOOOOE+OO 

0.7142860000 

3.4343400000 

166 

O.OOOOOOE+OO 

0.1052630000 

1.0000000000 

167 

O.OOOOOOE+OO 

0.2500000000 

0.9384840000 

168 

O.OOOOOOE+OO 

-0.1219510000 

1 .0000000000 

169 

O.OOOOOOE+OO 

-0 . 1600000000 

1 .0000000000 

170 

O.OOOOOOE+OO 

-0.2000000000 

0.3179990000 

191 

O.OOOOOOE+OO 

0.1578950000 

1.8822400000 

192 

O.OOOOOOE+OO 

1.0000000000 

5.7481300000 

193 

O.OOOOOOE+OO 

1.0000000000 

5.9166000000 

194 

O.OOOOOOE+OO 

1 . 0000000000 

5.4109700000 

195 

O.OOOOOOE+OO 

1 .0000000000 

5.9662100000 

201 

O.OOOOOOE+OO 

-0.1428570000 

1.1151800000 

202 

O.OOOOOOE+OO 

-0.8571430000 

1 . 5666600000 

203 

O.OOOOOOE+OO 

-1 .0000000000 

1 . 5640100000 

204 

O.OOOOOOE+OO 

-0.6000000000 

2.0986200000 

205 

O.OOOOOOE+OO 

-0.5555560000 

1.4384500000 

206 

O.OOOOOOE+OO 

0.4000000000 

1.7214400000 

207 

O.OOOOOOE+OO 

-0.7692 3 IE-01 

0.3186420000 

208 

O.OOOOOOE+OO 

0.2105260000 

0.3987230000 

209 

O.OOOOOOE+OO 

-0.1951220000 

1.0000000000 

210 

O.OOOOOOE+OO 

-0.6111110000 

1.2574600000 

211 

O.OOOOOOE+OO 

0.5789470000 

4.0343700000 

212 

O.OOOOOOE+OO 

1.0000000000 

4.2231000000 

213 

O.OOOOOOE+OO 

0.1250000000 

1.4560000000 

214 

O.OOOOOOE+OO 

0. 750000E-01 

0.1280370000 

215 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

O.OOOOOOE+OO 

216 

O.OOOOOOE+OO 

1.0000000000 

5.8410200000 

217 

O.OOOOOOE+OO 

1.0000000000 

5.8369600000 

218 

O.OOOOOOE+OO 

0.5714290000 

4.0051300000 

219 

O.OOOOOOE+OO 

0.2708330000 

1.8372700000 

220 

O.OOOOOOE+OO 

0.5000000000 

1.7378500000 

221 

O.OOOOOOE+OO 

1.0000000000 

5.5753100000 

222 

O.OOOOOOE+OO 

1 . 0000000000 

4.8800700000 

223 

O.OOOOOOE+OO 

0.6521740000 

3.9919100000 

224 

O.OOOOOOE+OO 

0.2881360000 

2.2592000000 

225 

O.OOOOOOE+OO 

0.3750000000 

1.9596100000 

226 

O.OOOOOOE+OO 

1.0000000000 

4.4994000000 

227 

O.OOOOOOE+OO 

1.0000000000 

5.3406700000 

228 

O.OOOOOOE+OO 

0. 5000000000 

4.0873900000 
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TABLE  C-4.  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  NEUTRAL  SEGMENT  (Continued) 

GROUP  1  TERMINATED  DEFENSIVE  GROUP  2  TERMINATED  OFFENSIVE 


Measure 

Pooled  Minimum 

Pooled  Range 

_t 

229 

0 . 000000E+00 

1 . 0000000000 

2.0815400000 

230 

0 . 000000E+00 

0.7368420000 

1.8814000000 

231 

0.000000E+00 

0.4545450000 

0.5751260000 

232 

O.OOOOOOE+OO 

-0.6250000000 

0.8496060000 

233 

0 . 000000E+00 

0.4736840000 

0.526746E-01 

234 

0.000000E+00 

-0.7222220000 

0.6590380000 

235 

O.OOOOOOE+OO 

-0.5454550000 

1.0138200000 

236 

O.OOOOOOE+OO 

-0.4210530000 

2.2726500000 

237 

O.OOOOOOE+OO 

-1 . 0000000000 

6.3317300000 

238 

O.OOOOOOE+OO 

-1.0000000000 

6.3152400000 

239 

O.OOOOOOE+OO 

-1 . 0000000000 

5.6427700000 

240 

O.OOOOOOE+OO 

-1.0000000000 

6.4517700000 

246 

O.OOOOOOE+OO 

-1.0000000000 

1.1922000000 

247 

O.OOOOOOE+OO 

-0.5909090000 

1.1001000000 

248 

O.OOOOOOE+OO 

-0.5714290000 

1.3618900000 

249 

O.OOOOOOE+OO 

-0.2195120000 

1.0000000000 

250 

O.OOOOOOE+OO 

-0.2777780000 

1.4132400000 

271 

O.OOOOOOE+OO 

-1.0000000000 

6.7717600000 

272 

O.OOOOOOE+OO 

-1 .0000000000 

5.S074300000 

273 

O.OOOOOOE+OO 

-0.5277780000 

4.3266000000 

274 

O.OOOOOOE+OO 

-1.0000000000 

2.2978400000 

275 

O.OOOOOOE+OO 

-0.7368420000 

1.6528000000 

311 

O.OOOOOOE+OO 

-1.0000000000 

4.5341700000 

312 

O.OOOOOOE+OO 

-1.0000000000 

4.2763400000 

313 

O.OOOOOOE+OO 

-0.4090910000 

3 . 5539200000 

314 

O.OOOOOOE+OO 

-0.3846150000 

2.0246800000 

315 

O.OOOOOOE+OO 

-0.3750000000 

1.3867900000 
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TABLE  C-5 .  UNIT-SCALED  SEGMENT  SUMMARY  MODEL  FOR  OFFENSIVE  SEGMENT 
GROUP  1  LOST  ADVANTAGE  GROUP  2  FIRED 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

3 

153.9490000000 

578.126000 

0.829626 

5 

0.1246130000 

-  72.111800 

1.134830 

6 

0.289449E-02 

0.277239 

1.228660 

8 

0.6108200000 

-  66.762400 

1.060230 

11 

1.0979200000 

-  88.582000 

1.278420 

12 

0.1838210000 

30.022300 

1.541350 

13 

-298.4940000000 

-1003.250000 

0.984364 

14 

20.5588000000 

-  707.104000 

2.270160 

15 

15.5030000000 

-  690.250000 

1.774900 

17 

0.1650960000 

-  76.044100 

1.069080 

23 

0.5399630000 

-  77.899700 

0.891716 

24 

0.7207590000 

-  69.519000 

0.838816 

28 

-  37.1925000000 

-  68.427500 

1.267460 

30 

0.5666530000 

47.060700 

0.889184 

Statistics  on  new 

variable : 

Mean 

Std  Dev 

Grpl 

-1.99588 

1.34911 

Grp2 

-2.47057 

1.49429 

Pooled 

-2.27971 

1.45296 

Break  Point  =  0.3970 

FI:  5.10985 

F2 :  748.09200 


SSB:  10.5092 

SSW:  396.9340 
SST:  407.4430 

Wi Ikes -Lambda :  0.974207 

S~ Omega:  0. 20641 IE-01 

R-Squared:  0.257930E-01 


Predicted  Min.  Percent  Error 


39.18 


TABLE  C-6 .  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  OFFENSIVE  SEGMENT 
GROUP  1  LOST  ADVANTAGE  GROUP  2  FIRED 


ure 

Pooled  Minimum 

Pooled  Range 

t 

171 

0.000000E+00 

1 . 000000 

1.886750 

172 

0.000000E+00 

0.125000 

1.000000 

175 

O.OOOOOOE+OO 

-0.875000 

1 . 000000 

177 

O.OOOOOOE+OO 

-1.000000 

0.961937 

178 

O.OOOOOOE+OO 

-1 . 000000 

1.335630 

180 

O.OOOOOOE+OO 

-1.000000 

1.117510 

182 

0.000000E+O< 

1.000000 

1.696880 

185 

O.OOOOOOE+OO 

-1.000000 

1.225960 

188 

0.000000E-K)0 

1.000000 

0.934416 

189 

0.000000E-H30 

-1.000000 

1.627500 

190 

O.OOOOOOE+OO 

1.000000 

1.000000 

Statistics 

on  new  variable: 

Mean 

Std  Dev 

Grpl 

0.56263 

0.75822 

Grp2 

-0.08227 

0.85078 

Pooled 

0.17326 

0.87134 

Break 

Point  =  1.0000 

FI 

:  33.7700 

F2:  4309.0400 

SSB:  19.3998 
SSW:  60.3190 
SST:  79.7188 

Wilkes-Lambda :  0.756648 
S-Omega:  0.234457 
R-Squared:  0.243352 

Predicted  Min.  Percent  Error  =  17.92 
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TABLF  1  7 .  UNIT  SCALED  SEGMENT  SUMMARY  MODEL 
POR  DEFENSIVE  SEGMENT 
GROrp  1  ESCAPED  GROUP  2  WAS  FIRED  AT 


Measure 

Pooled  Minimum 

Pooled  Range 

t_ 

12 

126.617000 

52.5588 

0.959819 

13 

-298.494000 

-1003.2500 

0.998035 

14 

23.055000 

-  756.6720 

2.051270 

15 

15.503000 

-  690.2500 

1.787550 

16 

1.000000 

3.0000 

1.409040 

18 

1.000000 

3.0000 

1.396640 

20 

4.615220 

-1352.1500 

0.995522 

21 

0.206214 

-  430.6020 

0.826603 

22 

-  71.965900 

-  148.8110 

0.976289 

23 

0.620147 

-  85.1120 

1.141590 

28 

-  51.981600 

218.7470 

0.849673 

29 

0.732483 

-  251.6340 

2.006210 

Statistics  on  new 

variable : 

Mean 

Std  Dev 

Grpl 

1.23982 

0.50624 

Grp2 

0.90457 

0.78061 

Pooled 

1.03936 

0.70178 

Break  Point  *  1.5475 

FI:  11.2650 

F2:  1747.1300 

SSB:  5.24198 

SSW:  89.80950 
SST:  95.05140 


Wilkes-Lambda : 
S -Omega : 
R-Squared: 


0.944851 
0. 500085E-01 
0.551489E-01 


Predicted  Min.  Percent  Error  >  50.0 
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TABLE  C-8.  UNIT  SCALED  TACSPACE  DISCRIMINANT  MODEL 
FOR  DEFENSIVE  SEGMENT 
GROUP  1  ESCAPED  GROUP  2  WAS  FIRED  AT 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

76 

O.OOOOOOE+OO 

1 . 000000 

2.512900 

79 

0 . 000000E+00 

-0.875000 

0.926216 

80 

O.OOOOOOE+OO 

-0.875000 

1.239090 

279 

O.OOOOOOE+OO 

-1 . 000000 

1.605180 

317 

O.OOOOOOE+OO 

1.000000 

1.863580 

318 

0.000000E+00 

-1.000000 

1.225570 

320 

O.OOOOOOE+OO 

-1.000000 

1.377280 

Statistics  on  new 

variable : 

Mean 

Std  Dev 

Grpl 

0.36811 

0.60342 

Grp2 

-0.23634 

0.63795 

Pooled 

0.01299 

0.68857 

Break  Point  ■  0.6250 

FI:  65.9900 
F2 :  6375.6300 

SSB:  17.0476 
SSW:  20.4086 
SST:  37.4562 

Wilkes-Lambda :  0.544865 
S-Oraega:  0.445167 
R-Squared:  0.455135 


Predicted  Min 


Percent  Error  =  26.25 


TABLE  C-9 .  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  SUMMARY  SEGMENT  MODEL 
FOR  REDUCED  ACMI /TACTS  WHOLE  ENGAGMENT 


ire 

Pooled  Minimum 

Pooled  Range 

t 

1 

256.0700000000 

-409.744000 

3.35441 

2 

369.7940000000 

-450.905000 

5.55015 

3 

262 . 5890000000 

-417.787000 

3.59137 

6 

0.940246E-02 

0.195849 

4.65563 

7 

-  5.6554900000 

5.655490 

1.97343 

9 

1.0310700000 

-  4.632270 

3.04957 

10 

-124.5940000000 

-242.841000 

1.19951 

11 

131.3440000000 

-228.627000 

1.79246 

12 

19.1357000000 

-152.309000 

9.00980 

25 

0.000000E+00 

0.883333 

4.28832 

26 

O.OOOOOOE+OO 

-  1.000000 

7.95391 

29 

17.4137000000 

935.399000 

2.34531 

30 

3.3577100000 

82.197200 

4.06604 

Mean 

Std  Dev 

Grpl 

-1.29783 

1.07531 

Grp2 

-4.02234 

0.85623 

Pooled 

-2.61143 

1.67997 

FIs  110.976 

F2:  2864.000 

Break  Point  ®  -3. 

3424 

SSB: 

103.7890 

SSW: 

51.4378 

SST: 

155.2260 

Wilkes 

-Lambda:  0.331373 

S-Omega:  0.658634 

R-Squared:  0.668627 

Predicted  Min.  Precent  Error  =  7.14 
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TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT 


ire 

Pooled  Minimum 

Pooled  Range 

1 

O.OOOOOOE+OO 

-0 . 145455E-01 

1.362060 

2 

O.OOOOOOE+OO 

-0 . 176991E-01 

0.861799 

3 

O.OOOOOOE+OO 

-0.221239E-02 

1.430230 

4 

O.OOOOOOE+OO 

-0.199115E-01 

1.512840 

5 

O.OOOOOOE+OO 

-0.1552630000 

3.661780 

7 

O.OOOOOOE+OO 

0.248756E-02 

1.795020 

11 

O.OOOOOOE+OO 

-0 . 110497E-01 

0.984895 

14 

O.OOOOOOE+OO 

0.442478E-02 

1.726860 

15 

O.OOOOOOE+OO 

0. 116279E-01 

1.563640 

18 

O.OOOOOOE+OO 

0.118694E-01 

1.303780 

21 

O.OOOOOOE+OO 

0. 101010E-01 

1.377550 

22 

O.OOOOOOE+OO 

-0 . 141509E-01 

1.579220 

23 

O.OOOOOOE+OO 

-0.110497E-01 

1.840810 

24 

O.OOOOOOE+OO 

-0.524194E-01 

1.292090 

25 

O.OOOOOOE+OO 

-0.29 761 9E-01 

2.128630 

28 

O.OOOOOOE+OO 

-0.298507E-01 

1.825160 

29 

O.OOOOOOE+OO 

-0. 22181 IE-01 

3.316770 

30 

O.OOOOOOE+OO 

-0.578778E-01 

4.076410 

31 

O.OOOOOOE+OO 

-0.436364E-01 

1.403670 

32 

O.OOOOOOE+OO 

-0 . 545455E-01 

1.488550 

33 

O.OOOOOOE+OO 

-0.301370E-01 

1.857840 

34 

O.OOOOOOE+OO 

0.468384E-02 

1.000000 

35 

O.OOOOOOE+OO 

-0.2611940000 

2.970870 

36 

O.OOOOOOE+OO 

-0.377907E-01 

1.131160 

37 

O.OOOOOOE+OO 

-0.763636E-01 

1.773720 

39 

O.OOOOOOE+OO 

-0 . 316456E-01 

1.835830 

40 

O.OOOOOOE+OO 

-0.1740330000 

5.451510 

42 

O.OOOOOOE+OO 

0. 497512E-02 

1.000000 

45 

O.OOOOOOE+OO 

-0 . 307329E-01 

2.692280 

48 

O.OOOOOOE+OO 

0. 467290E-02 

1.000000 

51 

O.OOOOOOE+OO 

0.890208E-02 

1.044950 

52 

O.OOOOOOE+OO 

0. 234192E-02 

1.000000 

53 

O.OOOOOOE+OO 

0. 290698E-02 

1.000000 

55 

O.OOOOOOE+OO 

0. 290698E-02 

1.000000 

57 

O.OOOOOOE+OO 

-0.118694E-01 

1.549500 

58 

O.OOOOOOE+OO 

-0 . 118694E-01 

1.707160 

59 

O.OOOOOOE+OC 

-0.145349E-01 

1.958310 

63 

O.OOOOOOE+OO 

-0.943396E-02 

2.895700 

64 

O.OOOOOOE+OO 

-0 . 936768E-02 

2.943990 

65 

O.OOOOOOE+OO 

-0.712166E-01 

2.017270 

66 

O.OOOOOOE+OO 

-0 . 298507E-01 

1.932420 

67 

O.OOOOOOE+OO 

-0.149254E-01 

1.219140 
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TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

69 

O.OOOOOOE+OO 

-0.376569E-01 

1.904930 

70 

O.OOOOOOE+OO 

-0.763158E-01 

3.249170 

71 

O.OOOOOOE-fOO 

-0 . 547945E-01 

2.251530 

72 

O.OOOOOOE+OO 

-0.526316E-01 

2.215990 

73 

O.OOOOOOE-fOO 

-0.491400E-01 

1.338260 

74 

O.OOOOOOE+OO 

0.210773E-01 

1.000000 

75 

O.OOOOOOE-fOO 

-0.2698410000 

1.859450 

76 

O.OOOOOOE-fOO 

-0 . 872093E-01 

1.555110 

77 

O.OOOOOOE-fOO 

-0.690909E-01 

1.251520 

78 

O.OOOOOOE-fOO 

0.2833330000 

0.934640 

79 

O.OOOOOOE-fOO 

-0.278638E-01 

1.000000 

80 

O.OOOOOOE-fOO 

-0.1546960000 

1.796790 

82 

O.OOOOOOE-fOO 

-0 . 593472E-02 

1.000000 

84 

O.OOOOOOE+OO 

-0 . 148368E-01 

1.000000 

85 

O.OOOOOOE+OO 

-0.278638E-01 

1.960050 

87 

O.OOOOOOE-fOO 

0.316456E-02 

1.000000 

96 

O.OOOOOOE-fOO 

-0.483092E-02 

1.000000 

97 

O.OOOOOOE-fOO 

-0. 70093 5E-02 

1.762300 

101 

O.OOOOOOE-fOO 

-0.221239E-02 

1.000000 

103 

O.OOOOOOE-fOO 

-0 . 143369E-01 

1.000000 

106 

O.OOOOOOE+OO 

-0. 148368E-01 

2.005910 

107 

O.OOOOOOE+OO 

-0.373134E-01 

0.974081 

113 

O.OOOOOOE-fOO 

-0.234192E-02 

1.000000 

114 

O.OOOOOOE+OO 

0.327103E-01 

1.000000 

115 

O.OOOOOOE+OO 

0.267009E-01 

1.480820 

116 

O.OOOOOOE+OO 

0.793651E-02 

0.892433 

117 

O.OOOOOOE+OO 

-0 . 136986E-01 

1.000000 

118 

O.OOOOOOE+OO 

0.1666670000 

1.073950 

119 

O.OOOOOOE-fOO 

0. 666667E-01 

0.961215 

121 

O.OOOOOOE-fOO 

0. 390208E-02 

1.054590 

126 

O.OOOOOOE-fOO 

-0 . 298736E-02 

1.436230 

127 

O.OOOOOOE+OO 

0.890208E-02 

1.000000 

129 

O.OOOOOOE+OO 

0. 148368E-01 

1.000000 

130 

O.OOOOOOE-fOO 

0.  247678E-01 

1.969020 

135 

O.OOOOOOE+OO 

0. 165485E-01 

1.918430 

136 

O.OOOOOOE-fOO 

0.436364E-01 

1.157010 

138 

O.OOOOOOE+OO 

0. 516351E-02 

1.330330 

139 

O.OOOOOOE+OO 

0. 943396E-02 

1.741510 

140 

O.OOOOOOE+OO 

0.1552630000 

3.868050 

141 

O.OOOOOOE+OO 

0.218182E-01 

1.000000 

142 

O.OOOOOOE+OO 

0.232558E-01 

1.347900 

143 

O.OOOOOOE+OO 

0.166667E-01 

2.166330 

144 

O.OOOOOOE+OO 

0.333333E-01 

2.211800 
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TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

145 

0 . OOOOOOE+OO 

0.1548120000 

3.727300 

148 

0.000000E+00 

0.184211E-01 

1.821710 

149 

0. 000000E+00 

0. 171990E-01 

2.199110 

150 

0. 000000E+00 

0.1088710000 

3.268250 

152 

0. 000000E+00 

0. 890208E-02 

2.861320 

153 

0. 000000E+00 

0. 144928E-01 

3.459250 

154 

0. OOOOOOE+OO 

0. 298507E-01 

4.012070 

155 

0. OOOOOOE+OO 

0. 604839E-01 

3.316720 

156 

0. OOOOOOE+OO 

0. 158720E-01 

3.635970 

157 

0. OOOOOOE+OO 

0. 122850E-01 

3.676340 

158 

0. OOOOOOE+OO 

0. 283019E-01 

2.439130 

159 

0. OOOOOOE+OO 

0. 201613E-01 

1.149540 

161 

0. OOOOOOE+OO 

-0.890208E-02 

3.346460 

162 

0. OOOOOOE+OO 

-0 . 148368E-01 

3.506700 

163 

0. OOOOOOE+OO 

-0. 28301 9E-01 

2.697070 

164 

0. OOOOOOE+OO 

-0.201613E-01 

1.890800 

165 

0. OOOOOOE+OO 

-0.425139E-01 

0.928661 

166 

0. OOOOOOE+OO 

-0 . 793651E-02 

1.267300 

167 

0. OOOOOOE+OO 

0.136986E-01 

1.000000 

168 

0. OOOOOOE+OO 

0.1166670000 

0.923342 

169 

0. OOOOOOE+OO 

0.666667E-01 

1.070940 

170 

0. OOOOOOE+OO 

0.1333330000 

1.289450 

171 

0. OOOOOOE+OO 

0.821918E-02 

1.000000 

172 

0. OOOOOOE+OO 

0. 654545E-01 

1.594730 

173 

0. OOOOOOE+OO 

0.3333300000 

1.041230 

176 

0. OOOOOOE+OO 

0. 545455E-01 

1.239410 

177 

0. OOOOOOE+OO 

0.1236360000 

1.653630 

178 

0. OOOOOOE+OO 

0.333333E-01 

1.609960 

179 

0. OOOOOOE+OO 

0. 315456E-01 

1.910290 

180 

0. OOOOOOE+OO 

0.2170420000 

5.396040 

181 

0. OOOOOOE+OO 

0.1598840000 

1.457170 

182 

0. OOOOOOE+OO 

0.1135270000 

1.854460 

183 

0. OOOOOOE+OO 

0. 557276E-01 

2.069100 

184 

0. OOOOOOE+OO 

0.2333330000 

1.234460 

185 

0. OOOOOOE+OO 

0.2364560000 

5.346410 

186 

0. OOOOOOE+OO 

0. 967742E-01 

1.422590 

187 

0. OOOOOOE+OO 

0. 522388E-01 

0.991641 

189 

0. OOOOOOE+OO 

0.101010E-01 

0.826201 

190 

0. OOOOOOE+OO 

0.949367E-01 

3.377030 

191 

0. OOOOOOE+OO 

0. 301370E-01 

1.377610 

192 

0. OOOOOOE+OO 

0. 303030E-01 

1.463430 

195 

0. OOOOOOE+OO 

0. 467290E-01 

3.077560 

202 

0. OOOOOOE+OO 

-0.828729E-02 

2.968820 

79 


TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

203 

0.000000E+00 

204 

0.000000E+00 

205 

0.000000E+00 

208 

O.OOOOOOE+OO 

209 

O.OOOOOOE+OO 

210 

0.000000E+00 

211 

O.OOOOOOE+OO 

212 

O.OOOOOOE+OO 

213 

O.OOOOOOE+OO 

215 

O.OOOOOOE+OO 

216 

O.OOOOOOE+OO 

217 

O.OOOOOOE+OO 

218 

O.OOOOOOE+OO 

219 

O.OOOOOOE+OO 

220 

O.OOOOOOE+OO 

223 

O.OOOOOOE+OO 

225 

O.OOOOOOE+OO 

229 

O.OOOOOOE+OO 

234 

O.OOOOOOE+OO 

236 

O.OOOOOOE+OO 

237 

O.OOOOOOE+OO 

239 

O.OOOOOOE+OO 

240 

O.OOOOOOE+OO 

241 

O.OOOOOOE+OO 

242 

O.OOOOOOE+OO 

243 

O.OOOOOOE+OO 

244 

O.OOOOOOE+OO 

245 

O.OOOOOOE+OO 

246 

O.OOOOOOE+OO 

248 

O.OOOOOOE+OO 

251 

O.OOOOOOE+OO 

252 

O.OOOOOOE+OO 

253 

O.OOOOOOE+OO 

254 

O.OOOOOOE+OO 

255 

O.OOOOOOE+OO 

256 

O.OOOOOOE+OO 

258 

O.OOOOOOE+OO 

259 

O.OOOOOOE+OO 

260 

O.OOOOOOE+OO 

261 

O.OOOOOOE+OO 

262 

O.OOOOOOE+OO 

267 

O.OOOOOOE+OO 

269 

O.OOOOOOE+OO 

Pooled  Range 

t 

-0.165094E-01 

3.593380 

-0.298507E-01 

3.699120 

-0 . 685484E-01 

3.116920 

0. 468384E-01 

1.000000 

-0 . 327103E-01 

1.000000 

-0.257009E-01 

1.483170 

0. 298507E-01 

2.583640 

0.712074E-01 

1.959790 

0.540541E-01 

1.165020 

0.873016E-01 

1.851930 

0 . 690990E-01 

1.443600 

0. 552326E-01 

1.131820 

0. 309598E-01 

1.643460 

-0.936768E-02 

1.000000 

0.2611940000 

3.510130 

0.5882 3 5E-01 

0.925768 

0.1194030000 

3.383010 

0. 199115E-01 

1.869030 

o.ninioooo 

0.920598 

-0.438356E-01 

2.194160 

-0.348259E-01 

1.045500 

-0.298507E-01 

1.594810 

-0.476190E-01 

2.979440 

-0.789474E-01 

1.763550 

-0.203488E-01 

1.996620 

-0. 18421 IE-01 

1.900640 

-0. 171990E-01 

2.707050 

-0.1088710000 

3.829900 

0.148368E-01 

1.646118 

0. 296736E-02 

1.000000 

0. 298507E-01 

1.424150 

0. 149254E-01 

2.294820 

0.118694E-01 

2.125970 

0. 358423E-01 

1.181680 

0. 605263E-01 

2.223940 

-0 . 238095E-01 

1.150700 

0. 298507E-01 

2.260850 

0. 377358E-01 

3.309420 

0.660377E-01 

5.041140 

0.110619E-01 

0.871105 

0. 158730E-01 

0.938214 

0.597015E-01 

0.983142 

-0 . 277778E-01 

1.184920 
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TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT  (Continued) 


Measure 

Pooled  Minimum 

Pooled  Range 

t 

274 

0.000000E+00 

-0.187354E-01 

1.120110 

275 

0.000000E+00 

-0.1615040000 

2.583730 

276 

0. OOOOOOE+OO 

-0 . 967742E-01 

1.874110 

277 

0 . 000000E+00 

-0.581818E-01 

2.048410 

278 

0. 000000E+00 

-0.464396E-01 

1.220330 

279 

0 . OOOOOOE+OO 

-0.757576E-02 

1.532650 

280 

0. OOOOOOE+OO 

-0.1061090000 

3.916170 

281 

0.  OOOOOOE+OO 

-0 . 218182E-01 

1.240110 

285 

0. OOOOOOE+OO 

-0.1507940000 

3.296420 

287 

0. OOOOOOE+OO 

0.442478E-02 

1.000000 

288 

0. OOOOOOE+OO 

0. 358423E-02 

1.000000 

289 

0. OOOOOOE+OO 

0.358423E-02 

1.000000 

293 

0. OOOOOOE+OO 

0. 234192E-02 

1.000000 

294 

0. OOOOOOE+OO 

0.716846E-02 

1.327210 

295 

0. OOOOOOE+OO 

0. 265700E-01 

1,827730 

296 

0. OOOOOOE+OO 

-0 . 101010E-01 

2.279550 

298 

0. OOOOOOE+OO 

0. 892857E-02 

1.266240 

299 

0. OOOOOOE+OO 

0.362903E-01 

1.255140 

300 

0. OOOOOOE+OO 

0.766129E-01 

2.079070 

305 

0. OOOOOOE+OO 

-0.483871E-01 

1.656470 

306 

0. OOOOOOE+OO 

-0 . 165746E-01 

1.198810 

309 

0. OOOOOOE+OO 

0.257009E-01 

0.870949 

311 

0. OOOOOOE+OO 

-0.1018180000 

1.192970 

312 

0. OOOOOOE+OO 

-0.1236360000 

0.977326 

314 

0.000000E-*-Q0 

0. 234192E-02 

1 .000000 

315 

0. OOOOOOE+OO 

-0.1194030000 

3.389600 

316 

0. OOOOOOE+OO 

-0.472727E-01 

1.804900 

317 

0. OOOOOOE+OO 

-0.1236360000 

1.932730 

318 

0. OOOOOOE+OO 

-0.465396E-01 

0.932289 

320 

0. OOOOOOE+OO 

-0.3417720000 

5.888210 
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TABLE  C-10.  DISSIMILAR  AIRCRAFT,  UNIT-SCALED  TACSPACE 
DISCRIMINANT  MODEL  FOR  WHOLE  ENGAGEMENT  (Continued) 


Mean 

Grpl  4.66188 
Grp 2  -5.12831 
Pooled  -0.23322 


Break  Point  =  -1.0450 

FI:  275.996 

F2 :  32153.000 

SSB:  5607.10 

SSW:  4733.59 

SST:  10340.70 


Wilke8-Lambda :  0.457764 
S-Omega:  0.539212 
R-Squared:  0.542236 


Predicted  Min.  Percent  Error 


Std  Dev 

4.67598 

4.35224 

6.66188 


-  15.81 
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TABLE  D-l.  NEUTRAL  SEGMENT  SUMMARY  MEASURES:  MEANS  AND  STANDARD  DEVIATION 
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